Open Bug 1583858 (webext-commands-global) Opened 5 months ago Updated 12 days ago

[meta] Support Global Hotkeys in Web Extensions (commands.global)

Categories

(Core :: Widget, enhancement)

enhancement
Not set

Tracking

()

People

(Reporter: MeFisto94, Unassigned)

References

(Blocks 2 open bugs, )

Details

(Keywords: meta)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Currently Firefox only handles KeyboardEvents when the Window is in focus.
There are two use-cases which would profit from input handling when the browser window is unfocused:

  1. Web Extensions (see #1411795)
  2. Handling of the Media Keys (to support the MediaSession API (https://bugzilla.mozilla.org/show_bug.cgi?id=mediasession) and other related issues)

Note that handling of Use-Case 2 could be different from a Hotkey on the native-side of things (special Media APIs), but it would make sense to expose it as a regular hotkey on the internal API side.

First I would like to focus on case 1:
I already have the Windows side of things working, so one can listen to keypresses, mark them as "processed" or register a global hot key which is exclusive. I will append the code / methods soon (when I am on windows again)

For Linux I am struggleing with XGrabKey: It is returning 1 but no X11 Error Handler is thrown, even when calling XSetErrorHandler(NULL); right before the call (FF has it's own Handlers it seems). Pressing the key also doesn't work, but I don't know which Event Handler would receive this event.

I'll append to this bug as soon as I progress, but I'd appreciate any input/idea in the meantime.

Status: UNCONFIRMED → NEW
Component: Untriaged → Widget
Ever confirmed: true
Product: Firefox → Core

FYI, handling hardware media keys is tracking in bug1572869. Now we have implemented that in OSX first.

Oh yeah, good catch, yet another bug in there. For Linux, we have 1353652 and on Mac there is "Now Playing Info and registering for Remote Command Center actions.", did you already implement that?

Actually I am now also thinking that Global Hotkeys aren't the right way to go for Media Keys, however some web-extensions seem to have registered as a listener to them. So maybe we have to connect the systems a bit later-on. But I am certain now that Use Case 2 won't be relevant to this issue / handled separately.

(In reply to Marc Streckfuß from comment #2)

Oh yeah, good catch, yet another bug in there. For Linux, we have 1353652 and on Mac there is "Now Playing Info and registering for Remote Command Center actions.", did you already implement that?

Thank for your information! It sounds like a good idea to support MPRIS API. But I'm not quite sure what the Remote Command Center actions you meant?

Now in bug1251795, the author would like to use MediaPlayer framework to tell system what media is playing now, is is the same thing?

I think this is the correct framework, yes. I took that quote from apples site, there is a sample project for that which I just downloaded. There seems to be NowPlayingInfo to expose the information to the system and Remote Command Center to "remote control" the player, that is listening to play and pause. Actually I just saw a glimpse of "RemoteCommand" in the phabricator link, so I guess yes.
That's perfect so far, now I just need to see if windows supports such a thing as well, then we can completely isolate it from real hotkeys, apart from maybe injecting them as hotkeys as well, when they don't actually control the Media Session API

Depends on: 1584542

I think we can mark this bug as a meta bug for supporting receiving media hardware keys, even if Firefox is not on foreground. And use separate bugs to track the implementation of each platform.

Depends on: 1584030, 1591230
Summary: Support for Global Hotkeys → [meta] Support for Global Media Hotkeys

Except for keyboard events, I wonder how would the hardware headset events are handling? Are they going to be converted to certain keyboard keycodes and then we can use same mechanism to capture them?

It seems MediaPlayer framework on OSX could handle headset events as well, but I don't know how they would be handled on Windows and Linux. Any ideas?

Flags: needinfo?(masayuki)
Flags: needinfo?(marc.streckfuss)

I have seen headset "mute" buttons converted to keyboard events on Linux.

I would like to add that Chrome has had this for several years now:

http://developer.chrome.com/apps/commands#scope

And Firefox had it too, up until Firefox 57

(In reply to Steven Penny from comment #8)

I would like to add that Chrome has had this for several years now:

http://developer.chrome.com/apps/commands#scope

And Firefox had it too, up until Firefox 57

That is a different thing from what we will do in this bug, that issue is being tracked by bug1251795.

(In reply to Alastor Wu [:alwu] from comment #6)

Except for keyboard events, I wonder how would the hardware headset events are handling? Are they going to be converted to certain keyboard keycodes and then we can use same mechanism to capture them?

I'm not sure. I guess it depends on the implementation of utility of the device. I guess that posing WM_APP_COMMAND message to focused application is standard manner.

Flags: needinfo?(masayuki)

(In reply to Alastor Wu [:alwu] from comment #6)

Except for keyboard events, I wonder how would the hardware headset events are handling? Are they going to be converted to certain keyboard keycodes and then we can use same mechanism to capture them?

It seems MediaPlayer framework on OSX could handle headset events as well, but I don't know how they would be handled on Windows and Linux. Any ideas?

I think we make the mistake of thinking about Keys here at all. While Keys work when in foreground and there are ways to capture them in the background, this is only some "legacy" or "simple" way of solving things.

MPRIS for instance allows me to use the KDEConnect App on Linux, so I can control Media with my Smartphone, as long as I am in the same local network.
For Windows I'd suspect a headset maybe even having configurable buttons and then some software/driver decides what to do (thinking of gaming headsets, at least).

Both are things we don't need to worry about, as the OS should take care of it and translate it into the desired API.
For Windows: Since my keyboard triggers SMTC without any special knowledge, I guess the OS converts the keycode (as long as no one handles it?) to SMTC.
For Linux: I have no intel here what the Desktop Environment is doing, but I suspect they do it just the same. And if a certain headset doesn't react, then it's a thing between the headset and the DE.

So: Keys would be only relevant when the desired API isn't available, which means Windows < 8.1 or Desktop Environments without D-Bus or MPRIS Handling.

TBH I think we have a few bugs where we mix up keys and media api in general and even pulled in some bug about web extensions global shortcuts which is only related because the example web extension provides like a media session implementation.

Flags: needinfo?(marc.streckfuss)

Thank you for your explanation.

I totally misunderstood what MPRIS and SMTC can provide to us, I first thought SMTC is just providing a virtual control interface and didn't realize platforms (Windows&Linux) have already intergated media keys handling to SMTC and MPRIS. So the bug1591288 I filed was totally nosense, for each platform we should only need one media keys event source, which would uses MediaPlayer on OSX, MPRIS on Linux, SMTC on Windows and MediaController on Android (I don't know if it would handle headset media keys event though).

Therefore, in the meanwhile when you're implementing those native implementations, I would probably start to extend the interface of MediaControllerin order to support adding metadata and create other platform media keys event interfaces, in which you could easily add your native implementation for bug1353652 and bug1584501.

Depends on: 1353652

As Marc mentioned before, I incorrectly mixed the global hotkeys with the media keys control, but actually they are two different things, because for media keys we have other framework which directly supported and be integrated in platform, we have no need to monitor media keys directly.

Therefore, remove all those media keys related bugs dependency.

No longer blocks: media-control
No longer depends on: 1353652, 1584030, 1584542, 1591230
See Also: → media-control
Summary: [meta] Support for Global Media Hotkeys → Support for Global Hotkeys
Keywords: meta
Alias: webext-commands-global
Blocks: 1584542, 1411795
Summary: Support for Global Hotkeys → [meta] Support Global Hotkeys in Web Extensions (commands.global)

So, now that we've splitted Media Keys, this will be the Meta/Tracking Bug for adding the commands.global flag for Web-Extensions.
One usecase is outlined in 1411795 (I used this bug as tracking bug, since the other one is more focused on getting media keys to work, which would've been one usecase).
Another usecase would be the Riot WebExtension (https://addons.mozilla.org/de/firefox/addon/riot/) which needs that to support Push-To-Talk in voice calls, e.g.

On all platforms besides Windows, we cannot register a hotkey without being rejected by the OS when someone already claimed that hotkey.
Moreover commands can be changed by the user, so one question will be to determine how to propagate a failure.
My guess/recommendation would be to make a hint in the UI where one can change keybindings and maybe a notification on addon load when it fails (remember what hotkeys are available can be very dynamic, depending on the other applications) or maybe pass an event to the addon so it can decide how to notify the user, it might not be critical for its functioning.

I'm currently waiting on some directions for the windows implementation as there are multiple ways to get hotkeys.

You need to log in before you can comment on or make changes to this bug.