1615665 - the controllability of media keys is hijacking by firefox

Status: NEW

(Core :: Audio/Video: Playback, task, P3)

Description

[Alastor Wu [:alwu]]

As we would create media controller for a silent video as well, so playing a silent video can claim the audio focus on platform which allows us to use media keys to control those silent video.

However, we only destroy media controller when closing that tab, which says it one tab is constantly playing silent video, then the controllability of media keys would be always hijacking by that tab.

Comment 1

[Marc Streckfuß [:MeFisto94]]

Don't focus too much on the property of the videos being silent, the main problem here is:

• The User is listening to music in their music player
• The User is browsing a media-heavy site like reddit, 9gag, some news papers
• The User presses Media-Keys in an intent to control the music player, but the browser reacts by pausing one of the contents of the media-heavy site.

Note: this is especially a problem when the side is continuing to start new media, because there, we might regain the OS Media Control focus every time new media starts playing.

So what needs to be solved is: How can we determine what kind of media does the user want to control and which media should be "background" stuff.

Another interesting use-case are notifications with a short "bing" sound. They probably shouldn't get focus at all, but if they do, they should drop the focus again, we don't want to "re-start" them.

How can the foreground/main media content be determined? Sound is probably a good category but probably not sufficient (think: videos with an audio track with the volume turned down to zero or almost zero, think: videos with a silent audio track).
Also: Could there be a case where a muted video should be paused? I guess so, but I can't come up with an example.

Comment 2

[Alastor Wu [:alwu]]

Background is one point, but audible or not is also something we have to consider. Should we really want to control an muted video by media key? Like, Facebook usually plays inaudible media, which is not "background" stuff, but I would feel that user would more likely to control background music, not facebook video.

It's hard to come out with a best solution for all situations, because I can think of too many edge cases.

FYI, Chrome's behavior is that, if media starts without sound, then the media key won't be intercepted, but once that media becomes audible, they would start intercepting media key even if the media become inaudible again.

Comment 3

[Alastor Wu [:alwu]]

(In reply to Marc Streckfuß [:MeFisto94] from comment #1)

Note: this is especially a problem when the side is continuing to start new media, because there, we might regain the OS Media Control focus every time new media starts playing.

But if media is audible, then it makes sense to control them by using media control, isn't it?

Comment 4

[Alastor Wu [:alwu]]

I will leave this bug as a place where we can discuss what best behaviors (more about UX feeling) we should have, and file other bugs to do the actual fix.

Comment 5

[Marc Streckfuß [:MeFisto94]]

Today I came across two additional things, specifically here for notifications:

• Focus regain (i.e. in background there is a youtube video playing, then comes a notification sound, the notification sound timer expires, youtube should have focus again) and
• Focus Prevention / Filtering: Sounds < 3 seconds shouldn't be considered controllable.

Background: "When Hangouts is open and uses sound for notifications and you are not playing any other media, MPRIS will play the notification sound again"

Comment 6

[Alastor Wu [:alwu]]

(In reply to Marc Streckfuß [:MeFisto94] from comment #5)

Today I came across two additional things, specifically here for notifications:

• Focus regain (i.e. in background there is a youtube video playing, then comes a notification sound, the notification sound timer expires, youtube should have focus again) and

The timer is independent for each media element, so in this case, only the element used to play notification would lost the ability of being controlled.

• Focus Prevention / Filtering: Sounds < 3 seconds shouldn't be considered controllable.

Background: "When Hangouts is open and uses sound for notifications and you are not playing any other media, MPRIS will play the notification sound again"

That is considerable, but we have to consider that some media might provide wrong duration, which might be actually less than 3s, but its metadata would say its duration is longer than 3s. I would say for this case, timer seems enough, we just need to find how long would be the best time.

Adding more checks might just increase complexity but not gain equal return.

Comment 7

[Alastor Wu [:alwu]]

(In reply to Marc Streckfuß [:MeFisto94] from comment #5)

• Focus Prevention / Filtering: Sounds < 3 seconds shouldn't be considered controllable.

After some testing, I found Chrome has already prevented to intercept media key when they play short sound, but I don't sure what criteria they use. Anyway, let's use timer first and see if it can achieve good user experience or we have to add another check for short sound.

Comment 8

[Marco Bonardo [:mak]]

I was about to file a bug, but apparently it may be covered by this one.
I had to disable media.hardwaremediakeys.enabled because when trying to control my media player in background (MusicBee), it was additionally showing a huge Windows Volume OSD with [Mozilla Firefox] and controls, covering most of the top left part of the browser window, and playing sounds from a pinned Slack tab.
Please let me know if you need more details or want me to file something apart.

Comment 9

[Alastor Wu [:alwu]]

(In reply to Marco Bonardo [:mak] from comment #8)

I was about to file a bug, but apparently it may be covered by this one.
I had to disable media.hardwaremediakeys.enabled because when trying to control my media player in background (MusicBee), it was additionally showing a huge Windows Volume OSD with [Mozilla Firefox] and controls, covering most of the top left part of the browser window, and playing sounds from a pinned Slack tab.
Please let me know if you need more details or want me to file something apart.

Sorry for the inconvenience, now we would fix it in bug 1617033. Because of this, I think using timer is not enough, because it can't prevent showing the SMTC notification on Windows, so I would also handle this issue in bug1617033.

Comment 10

[Marc Streckfuß [:MeFisto94]]

Attachment: SMTC-Switch-Arrow.png

Comment 11

[Marc Streckfuß [:MeFisto94]]

Not a real solution, but see the attached file, the arrow to the right should enable you to switch to your real music player.
It could be that you have to do it after every slack message, which is why we are working on trying to fix this problem.

If you have any idea on algorithms or heuristics on which media shouldn't be controllable, feel free to comment that here.

Comment 12

[Marco Bonardo [:mak]]

MusicBee doesn't register for that OSD, it doesn't pop it up, and I didn't have an arrow to switch to it.
Even if it'd allow to switch, I wouldn't use those OSD controls, because they pop up when I use media keys on the keyboard, so why should I then click on those OSD buttons? I hope MS will re-valuate this UI or allow an option to disable it.

Comment 13

[Marc Streckfuß [:MeFisto94]]

MusicBee doesn't register for that OSD, it doesn't pop it up, and I didn't have an arrow to switch to it.
This is an interesting detail, because it means that MusicBee either uses an older Windows API or listens to Media Keys directly and thus isn't part of the "Audio Focus Pool", where the OS tries to only redirect the key presses to the correct application.

Actually I never thought about the fact that these buttons don't make much sense as they are mostly toggled by keyboard keys, but you are right about that :D But apart from that it is probably going to stay, it just looks bad because we don't provide an album cover and media titles, yet.