1667454 - Add a toggle button on `about:preference` for media control

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Alastor Wu [:alwu]]

It would be easier to have a toggle button to enable/disable media-control, rather than modifying prefs in about:config.

Comment 1

[Alastor Wu [:alwu]]

Attachment: Bug 1667454 - part1 : add a toggle button in preference page for media control.

Comment 2

[Alastor Wu [:alwu]]

Attachment: Bug 1667454 - part2 : add a test.

Comment 3

[Alastor Wu [:alwu]]

Attachment: Bug 1667454 - part3 : modify test.

Comment 4

[Alastor Wu [:alwu]]

Reply Gij's comment [1] here in order to let UX/product join the discussion.

Please talk to UX/product about this change, where the UI should live, what wording to use, and if/how to integrate it. For any new pref/option X, it's always easier to add them to the preferences/options than to manipulate them in about:config - until there's 500 items in the list and it's impossible to find things.

This idea is originated from a discussion with adam (product manager) because we've seen some posts on Reddits where people ask how to disable media control since the feature has shipped on Fx81. We both think having a toggle button on preference page would be easier than asking user to adjuct their preference. Because this feature would impact all users who watch media on Firefox, which is quite large amount, so we believe having that toggle button is an important thing.

Do we have telemetry how many people even have machines / devices with media keys? Do we support those keys everywhere? Perhaps we could only show this if the machine has the relevant keys?

No, we don't have that information, which is also a hard-measure data, I don't know if it's possible to know if the keyboard users use having media key or not. (but we're going to add some pings in bug1668139 later) But for machines, according to data on Sep 20 [2], it's around 65% users have a machine which supports media control.

However, media control feature is avaliable on almost all desktop firefox [3], no matter they have media keys or not, because on some platforms, they would provide a virtual control interface for users. On MacOS, all keyboards have media keys. On Windows and Linux, even if users don't have keyboard with media keys, they can still control media via virtual control interface.

In addition, that features is not only supporting keyborad, it also supports hardware media keys on headset, which is avaliable on all platforms. [4]

Perhaps we could only show this if the machine has the relevant keys? Yes, we can do so for those unsupported platforms (eg. Windows 7)

Does this really need its own heading, and if so, should we move the autoplay controls under it, too?

I don't think so (making it to autoplay control). This feature is completely different from autoplay control, making them together helps nothing.
But for heading, I don't have a strong opinion. It could be move to other group (eg. Browsing)

[1] https://phabricator.services.mozilla.com/D91871#2949246
[2] https://data.firefox.com/dashboard/hardware
[3] https://docs.google.com/document/d/1c4FivJpvAjjw9Uw-jn7X1UjGOoWkANXOulNyqDWs83w/edit#heading=h.2om17cr6dvh
[4] https://docs.google.com/document/d/1c4FivJpvAjjw9Uw-jn7X1UjGOoWkANXOulNyqDWs83w/edit#heading=h.oqqktkayxnbt

Comment 5

[Alastor Wu [:alwu]]

Hi, Adam,
Could you help us ask UX to involve in this discussion as well? Thank you.

Comment 6

[:Gijs (he/him)]

(In reply to Alastor Wu [:alwu] from comment #4)

Reply Gij's comment [1] here in order to let UX/product join the discussion.

Please talk to UX/product about this change, where the UI should live, what wording to use, and if/how to integrate it. For any new pref/option X, it's always easier to add them to the preferences/options than to manipulate them in about:config - until there's 500 items in the list and it's impossible to find things.

This idea is originated from a discussion with adam (product manager) because we've seen some posts on Reddits where people ask how to disable media control since the feature has shipped on Fx81.

How common is wanting to disable it, and what's the motivation for doing so? Does the feature not work in some cases? Is it just about conflicts with other applications? Basically, is there a possibility to fix the complaints without relying on users to find the toggle in the options / preferences (which a non-trivial number of users won't) ?

What do other browsers do? I can't find an option for this functionality in Chrome, for instance...

Perhaps we could show a doorhanger notification the first time we intercept a key and handle it and allow users to turn it off then, and only show the checkbox in the options if they've been turned off (which is what we do for some other prefs), or something along those lines? It depends a bit what the reason is that people are wanting to turn this off.

We both think having a toggle button on preference page would be easier than asking user to adjuct their preference. Because this feature would impact all users who watch media on Firefox, which is quite large amount

This impacts people who use these keys to control that media (and/or potentially use those keys to control other apps' media, but don't want to use it for Firefox?). That's not really the same thing.

Do we have telemetry how many people even have machines / devices with media keys? Do we support those keys everywhere? Perhaps we could only show this if the machine has the relevant keys?

No, we don't have that information, which is also a hard-measure data, I don't know if it's possible to know if the keyboard users use having media key or not. (but we're going to add some pings in bug1668139 later) But for machines, according to data on Sep 20 [2], it's around 65% users have a machine which supports media control.

How do I find the 65% in the hardware report? Is it users on everything but Win7 and below, or something?

However, media control feature is avaliable on almost all desktop firefox [3], no matter they have media keys or not, because on some platforms, they would provide a virtual control interface for users. On MacOS, all keyboards have media keys.

My mac has a touchbar. The customizable set of Firefox controls does not include any media keys, and the touch bar doesn't respond to media starting to play / stopping to play like iTunes Music.app does. If I play music in both, the right-hand side of the touch bar shows an icon with a bar graph, which when tapped brings up controls for Music.app . None of these controls appear to affect Firefox. Only after I quit Music.app can I use the tiny [<] icon to bring up prev/pause/next controls that appear to then impact youtube. But that's... very laborious. Seems hard to imagine that's something that affects a lot of people -- or that they'd be desperate to turn it off, to be honest!

On Windows and Linux, even if users don't have keyboard with media keys, they can still control media via virtual control interface.

In addition, that features is not only supporting keyborad, it also supports hardware media keys on headset, which is avaliable on all platforms. [4]

Perhaps we could only show this if the machine has the relevant keys? Yes, we can do so for those unsupported platforms (eg. Windows 7)

Does this really need its own heading, and if so, should we move the autoplay controls under it, too?

I don't think so (making it to autoplay control). This feature is completely different from autoplay control, making them together helps nothing.
But for heading, I don't have a strong opinion. It could be move to other group (eg. Browsing)

I'm aware that this is different from the autoplay control, I'm saying, if we have a "Media" header (which is what your patch does), that's where I'd expect the autoplay controls to live, as they too are media-related...

Comment 7

[Alastor Wu [:alwu]]

(In reply to :Gijs (he/him) from comment #6)

How common is wanting to disable it, and what's the motivation for doing so? Does the feature not work in some cases? Is it just about conflicts with other applications? Basically, is there a possibility to fix the complaints without relying on users to find the toggle in the options / preferences (which a non-trivial number of users won't) ?

There are some related posts we collect from Reddit [1]. Some of them don't like the virtual contorl interface that is defined by Windows, and some of them don't use Firefox as a media player, so they don't want Firefox to intercept media keys from other applications (eg. Spotify)

The problem is that this feature works for those users who like to use Firefox as a media player so that they can easily control media without having more steps (finding a tab, finding a media, then do the operations), but doesn't work for those users don't treat Firefox as a media player.

The issues such like how the virtual control interface looks like, or how does Firefox intercept media keys from other applications, they are all defined by platforms. So we can't change those behaviors and each platfom has its own implementation.

[1] https://docs.google.com/document/d/1jW3o1XRs4FUHH3BW7vSoK3vGZVMTgw9Zq6H06Fpd2t4/edit?usp=sharing

What do other browsers do? I can't find an option for this functionality in Chrome, for instance...

If you search chrome media control on google, then you can see lots of related threads, which are probably people asking how to disable media control, or teaching people how to enable/disable that feature.

That started testing that on 2019 mid (or earlier, I didn't remember the exact date), and at that time we indeed saw lots of complaint about users want to disable Chrome's media control.

As I said above, this feature is really depend on how user treat their browser. So far Chrome still control that only behind the pref, which still results in lots of teaching posts online. For safari, it seems they didn't provide such an option so far [2].

[2] https://discussions.apple.com/thread/251619195

Perhaps we could show a doorhanger notification the first time we intercept a key and handle it and allow users to turn it off then, and only show the checkbox in the options if they've been turned off (which is what we do for some other prefs), or something along those lines? It depends a bit what the reason is that people are wanting to turn this off.

I don't agree with that. When I worked on blocking autoplay project, we've launched a study for the doorhanger, and the result is that almost no one would care about doorhanger.

This impacts people who use these keys to control that media (and/or potentially use those keys to control other apps' media, but don't want to use it for Firefox?). That's not really the same thing.

Not sure I understand what you mean here. I don't know if user would like to enable/disable media control multiple times, but one of the target users would be those who want to disable media control.

How do I find the 65% in the hardware report? Is it users on everything but Win7 and below, or something?

57.003 (Windows10) + 4.992 (Windows 8.1) + 4.992 (macOS Catalina) + 1.088 (macOS High Sierra) = 68.075 (oops, wrong result yesterday)

probably work (depend on what version) : 1.982 (Mac other) 1.658 (Linux 5.x) = 3.64

So if we consider those probably work, the result would be 71.715.

My mac has a touchbar. The customizable set of Firefox controls does not include any media keys, and the touch bar doesn't respond to media starting to play / stopping to play like iTunes Music.app does. If I play music in both, the right-hand side of the touch bar shows an icon with a bar graph, which when tapped brings up controls for Music.app . None of these controls appear to affect Firefox. Only after I quit Music.app can I use the tiny [<] icon to bring up prev/pause/next controls that appear to then impact youtube. But that's... very laborious. Seems hard to imagine that's something that affects a lot of people -- or that they'd be desperate to turn it off, to be honest!

That is another issue (bug 1660936) Firefox and Chrome doesn't have a customized UI for that (we use system defined control UI), but Safari does. And we can certainly do that if we want. In addition, you should also consider those Mac without touch bar, which has physical media keys.

On Windows8.1+ (~62%), users would all be available to use that feature even if they don't have keyboard with media key, because Windows would show virtual control interface.

I'm aware that this is different from the autoplay control, I'm saying, if we have a "Media" header (which is what your patch does), that's where I'd expect the autoplay controls to live, as they too are media-related...

Got it. But I don't have a strong opinion about if we need to add another new header. Adding a new header or putting on other headers (but still in about:preferences) both sounds good to me.

Comment 8

[Alastor Wu [:alwu]]

Here is the doc [1] created by UX.
[1] https://docs.google.com/document/d/1Faayu4_PnIxTDScIiOI3tu20_A5av3kryCRy26J2WyQ/edit

Comment 9

[:Gijs (he/him)]

(In reply to Alastor Wu [:alwu] from comment #7)

(In reply to :Gijs (he/him) from comment #6)

How common is wanting to disable it, and what's the motivation for doing so? Does the feature not work in some cases? Is it just about conflicts with other applications? Basically, is there a possibility to fix the complaints without relying on users to find the toggle in the options / preferences (which a non-trivial number of users won't) ?

There are some related posts we collect from Reddit [1]. Some of them don't like the virtual contorl interface that is defined by Windows, and some of them don't use Firefox as a media player, so they don't want Firefox to intercept media keys from other applications (eg. Spotify)

The problem is that this feature works for those users who like to use Firefox as a media player so that they can easily control media without having more steps (finding a tab, finding a media, then do the operations), but doesn't work for those users don't treat Firefox as a media player.

Should the pref be more than on/off (always use keys, only if in foreground, only "play" when in foreground, never ), and we could default to "only play if in foreground" ? Because most of these complaints seem to be about Firefox responding to the key when it's not in the foreground, but any old youtube tab is open anywhere in the background. I can understand the surprise of having a music app in the foreground and pressing "play" not being limited to that app...

Basically, I am resistant to "just add a (visible) pref that allows people to turn it off" as a solution, when it seems the problems go deeper than "I don't like any of this", and we're not solving much of the discoverability problem either. Even some of the complaining posts mention that they do like that they can do this, they just don't like it happening when they intended the key for some other app.

What is the "black box" that the reddit posts are mentioning?

Perhaps we could show a doorhanger notification the first time we intercept a key and handle it and allow users to turn it off then, and only show the checkbox in the options if they've been turned off (which is what we do for some other prefs), or something along those lines? It depends a bit what the reason is that people are wanting to turn this off.

I don't agree with that. When I worked on blocking autoplay project, we've launched a study for the doorhanger, and the result is that almost no one would care about doorhanger.

It's not clear to me what you used a doorhanger for and what "caring about the doorhanger" would mean here, but either way - I'm not attached to the doorhanger specifically; we could use an OS level notification (which would allow interaction so the user can undo play/pause if it's a mistake) or any other kind of contextual UI. The complaints speak of not being able to discover what causes the behaviour or how to control it, and adding the pref into the options helps a little bit but not a lot, whereas a contextual "this is what just happened, here's a checkbox to turn it off / don't show me again / whatever" would be better.

This impacts people who use these keys to control that media (and/or potentially use those keys to control other apps' media, but don't want to use it for Firefox?). That's not really the same thing.

Not sure I understand what you mean here.

I was responding to:

this feature would impact all users who watch media on Firefox, which is quite large amount

My point was not everyone who watches media has or uses hardware media keys, so you are overestimating who is impacted. Reddit is surely a vocal minority here.

Anyway, if we're convinced we need this, I can r+ a patch that implements the UX spec from the doc.

I'm still a little confused because the phrasing there says "hardware controls", and you said:

On Windows8.1+ (~62%), users would all be available to use that feature even if they don't have keyboard with media key, because Windows would show virtual control interface.

How does this interface come up, and is that really covered by "hardware controls" ?

Comment 10

[Alastor Wu [:alwu]]

(In reply to :Gijs (he/him) from comment #9)

Should the pref be more than on/off (always use keys, only if in foreground, only "play" when in foreground, never ), and we could default to "only play if in foreground" ? Because most of these complaints seem to be about Firefox responding to the key when it's not in the foreground, but any old youtube tab is open anywhere in the background. I can understand the surprise of having a music app in the foreground and pressing "play" not being limited to that app...

That is the platform's design, and you would encouter same situation when you're using two different media players. For example, you play Spotify first, then switch to another player (ex, VLC, if it implements media control) and play something, then you switch to Spotify, pressing control key would still control VLC which is owning the platform audio focus. So that is not something we can change.

In general, each platform would switch the platform audio focus (which is what app would receive media keys) to the latest application which requests for that (which depends on the app's implementation, but it usually happen when an app starts playing audio)

So if a user uses Firefox first, then switch to Spotify and start it via its app, then the Spotify would start receiving media keys again.

Controlling media even if Firefox is on background is one of our goal, and all applications uses same platform API would have same behavior.

Basically, I am resistant to "just add a (visible) pref that allows people to turn it off" as a solution, when it seems the problems go deeper than "I don't like any of this", and we're not solving much of the discoverability problem either. Even some of the complaining posts mention that they do like that they can do this, they just don't like it happening when they intended the key for some other app.

What is the "black box" that the reddit posts are mentioning?

In the meantime, I will add a telemetry in bug1668139 to measure how many users uses the toggle button to enable/disable the feature. If we really want to know how user satisfy with this feature, I think as least we should provide something and test, not just assume what the problem is.

I guess the post you mentions is that Spotify won't show the black box when it's in the foreground?

The black box is the virtual control interface showed in Windows, which shows the playing media metadata and some control buttons [1]. I've check the Windows API, and there is no such thing can do that, Chrome uses same API and they can't do the thing Spotify do as well. The only explanation I can make is that Spotify uses older Windows API to control the interface, and that API won't trigger showing the virtual control interface, but only affects the playback.

But we don't have a resource to implement the whole thing by using their old API, so currently I would prefer to make the implementation to as close as Chrome's, and then to improve our feature based on that.

[1] https://docs.microsoft.com/en-us/uwp/api/Windows.Media.SystemMediaTransportControls?view=winrt-19041

It's not clear to me what you used a doorhanger for and what "caring about the doorhanger" would mean here, but either way - I'm not attached to the doorhanger specifically; we could use an OS level notification (which would allow interaction so the user can undo play/pause if it's a mistake) or any other kind of contextual UI. The complaints speak of not being able to discover what causes the behaviour or how to control it, and adding the pref into the options helps a little bit but not a lot, whereas a contextual "this is what just happened, here's a checkbox to turn it off / don't show me again / whatever" would be better.

We used doorhanger to notice users that blocking autoplay is happening and tell users that they can go to some place to adjust the preference, but the result is that almost all users would just dismiss the doorhanger or ignore it. I agree that we should tell users what makes those thing (control playback via media keys) happen, probably need an official post? Another problem is that, this feature works even if Firefox is on the background, so showing the Firefox level message won't work at that situation.

What OS level notification we can use? Have we used that on other feature before?

My point was not everyone who watches media has or uses hardware media keys, so you are overestimating who is impacted. Reddit is surely a vocal minority here.

Make sense, so I'm thinking about adding a telemetry ping to see how many users enable the feature but don't use that to control media.

Anyway, if we're convinced we need this, I can r+ a patch that implements the UX spec from the doc.

Thank you, I will update my patch again after the UX folk confirms the wording and where we should put the toggle button.

I'm still a little confused because the phrasing there says "hardware controls", and you said:

On Windows8.1+ (~62%), users would all be available to use that feature even if they don't have keyboard with media key, because Windows would show virtual control interface.

How does this interface come up, and is that really covered by "hardware controls" ?

Sorry, I know the preference name is actually confusing. In the beginning when we were implementing that feature, we only focus on hardware media keys. But during that the process, we found out that the media keys also includes virtual one.

So another benefit of making a toggle button is that we can use proper wording and also be able to attach the SUMO link of media control on beside.

Comment 11

[Alastor Wu [:alwu]]

Attachment: Bug 1667454 - part2 : add telemetry probe to record the number of times a user changes media control setting.

Comment 12

[Alastor Wu [:alwu]]

Comment on attachment 9180333 [details]
Bug 1667454 - part2 : add telemetry probe to record the number of times a user changes media control setting.

Request for data collection review form

1. What questions will you answer with this data?

To know how many users would enable/disable the media control, and know how many users perform that via UI button.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

We want to know if this feature gets users happy or not, to know if we have a need to modify our feature.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

Currently we can only know that via some forum posts, which can't precisely represent all users' feeling. So we need to use Telemetry to know a whole picture.

4. Can current instrumentation answer these questions?

No, because the result (forum posts) is probably from a small portion of users, which can't precisely represent all users' feeling.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Probe name : MEDIA_CONTROL_SETTING_CHANGE
Categories : 1 & 2 (Technical & Interaction data)
Description : Counts the number of times a user enable or disable the media control feature via UI button, and the total number of times media control feature is enabled and disabled, which allow us to calculate the number of times a user changes setting via modifying the preference directly.

6. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is documented in its definitions files Histograms.json.

7. How long will this data be collected? Choose one of the following:

Until Fx90.

8. What populations will you measure?

On release and pre-release channels, all countried and locales.

9. If this data collection is default on, what is the opt-out mechanism for users?

That control by telemetry system, so I think user can disable Telemetry collection if they want.

10. Please provide a general description of how you will analyze this data.

To check the percentage of users disable the media control, if it's too high, then we would probably ask UX to review the whole use flow in order to improve that.

11. Where do you intend to share the results of your analysis?

That is a Telemetry probe, so everyone is free to check the result.

12. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

No

Comment 13

[Alastor Wu [:alwu]]

Comment on attachment 9180333 [details]
Bug 1667454 - part2 : add telemetry probe to record the number of times a user changes media control setting.

I will ask for a data review after finishing code review.

Comment 14

[Meridel [:meridel]]

Hi Flod, can you please review the proposed copy for Preferences and let me know if there are any issues from an l10n perspective? Thank you!
Location: Preferences > General > Browsing > Appears right after "Enable Picture-in-Picture video controls":

☑️ Control media via keyboard, headset, or virtual interface Learn more

Comment 15

[Francesco Lodolo [:flod]]

I don't see any specific issue for localization.

Do we have any example of "virtual interface"? Given the context, I assumed we were talking only about physical buttons.

Comment 16

[Meridel [:meridel]]

Thanks, Flod. There's a list of VI examples here: https://docs.google.com/document/d/1c4FivJpvAjjw9Uw-jn7X1UjGOoWkANXOulNyqDWs83w/edit#heading=h.oqqktkayxnbt

Comment 17

[Alastor Wu [:alwu]]

Attachment: data-review-request

Comment 18

[Chris H-C :chutten]

Comment on attachment 9181116 [details]
data-review-request

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in Firefox 90.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :alwu is responsible for renewing or removing the collection before it expires in Firefox 90.

---

Result: datareview+

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dfd0308f5ab7
part1 : add a toggle button in preference page for media control. r=preferences-reviewers,fluent-reviewers,Gijs
https://hg.mozilla.org/integration/autoland/rev/fe7de882fcbf
part2 : add telemetry probe to record the number of times a user changes media control setting. r=preferences-reviewers,mossop,Gijs

Comment 20

[Razvan Maries]

Backed out for perma failures.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=windows%2C7%2Copt%2Cmochitests%2Ctest-windows7-32%2Fopt-mochitest-browser-chrome-e10s%2Cbc1&revision=fe7de882fcbff03900c906d96006c444d4e0df16&selectedTaskRun=Prk0v4ycRIiLPw8tuzRdfQ.0

Logs:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318643652&repo=autoland&lineNumber=4381
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318626135&repo=autoland&lineNumber=4561

Backout: https://hg.mozilla.org/integration/autoland/rev/1aaadc2ea5340e8dbc689a62bc70d711b89b08c3

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d39c10220f60
part1 : add a toggle button in preference page for media control. r=preferences-reviewers,fluent-reviewers,Gijs
https://hg.mozilla.org/integration/autoland/rev/d0a79869a8c0
part2 : add telemetry probe to record the number of times a user changes media control setting. r=preferences-reviewers,mossop,Gijs

Comment 22

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/d39c10220f60
https://hg.mozilla.org/mozilla-central/rev/d0a79869a8c0

Comment 23

[Alastor Wu [:alwu]]

Release Note Request (optional, but appreciated)
[Why is this notable]: It introduce a new button on about:preference page to allow users easily enable/disable media control
[Affects Firefox for Android]: No
[Suggested wording]: No strong preference, but the wording we use for button is Control media via keyboard, headset, or virtual interface.
[Links (documentation, blog post, etc)]: We have a SUMO page for that, which would also appear alongside with the button.