Closed Bug 1643027 Opened 3 years ago Closed 2 years ago

[UX] The "Mute Mic" and "Mute Camera" buttons open the "Permissions" tab instead of muting the mic/camera

Categories

(Firefox :: Site Permissions, defect, P3)

Desktop
All
defect

Tracking

()

VERIFIED FIXED
82 Branch
Tracking Status
firefox79 --- disabled
firefox82 --- verified
firefox83 --- verified

People

(Reporter: mconley, Assigned: mconley)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

Attachments

(5 files)

[Affected Versions]:

  • Nightly 79.0a1
  • Firefox Beta 78.0b1
  • Firefox 77 RC-build 3

[Affected Platforms]:

  • All Windows;
  • All Mac;
  • All Linux;

[Prerequisites]:

  • The following preferences are set:
    -> privacy.webrtc.allowSilencingNotifications = true
    -> privacy.webrtc.legacyGlobalIndicator = false
  • Have at least 2 tabs opened.

[Steps to reproduce]:

  1. Open the browser with the profile from prerequisites.
  2. Start a webRTC video call on Zoom and share the entire screen.
  3. Click the "Mute Mic"/"Mute Camera" button from the Global Sharing Indicator.
  4. Observe the behavior.

[Expected results]:

  • The mic/camera is muted.

[Actual results]:

  • The "Permissions" tab is opened

[Notes]:

  • Attached a screen recording of the issue.

S1 or S2 bugs need an assignee - could you find someone for this bug?

Flags: needinfo?(pbz)
Assignee: nobody → mconley
Flags: needinfo?(pbz)

Opening the panel was an intentional decision at the time - this is essentially how the old indicator worked too. The buttons were never meant to let users mute or unmute their microphones or cameras - only to control the sharing of them.

Do you think there is anything left to do here, aaron? Or can I close this as WONTFIX?

Flags: needinfo?(abenson)

IIRC, having those buttons open up the permissions panel was a compromise made based on time constraints but I could have that wrong.

Either way, I think ideally those buttons would act as mute toggles. Having that functionality might even assuage some of the frustrations with the display covering up the underlying controls since they would be providing the similar functions.

Flags: needinfo?(abenson)

I apologize if I've asked this before, and have just forgotten... but does privileged JS have the ability to "mute" a stream being sent to a site?

Flags: needinfo?(jib)

(In reply to Aaron Benson from comment #3)

Either way, I think ideally those buttons would act as mute toggles.

Presumably, if the user is sharing to multiple sites, this is a global mute on microphone and camera inputs for all sites?

Flags: needinfo?(abenson)

(In reply to Mike Conley (:mconley) (:⚙️) (Extremely busy) from comment #5)

(In reply to Aaron Benson from comment #3)

Either way, I think ideally those buttons would act as mute toggles.

Presumably, if the user is sharing to multiple sites, this is a global mute on microphone and camera inputs for all sites?

Yeah, I would think so.

Flags: needinfo?(abenson)

(In reply to Mike Conley (:mconley) (:⚙️) (Extremely busy) from comment #4)

I apologize if I've asked this before, and have just forgotten... but does privileged JS have the ability to "mute" a stream being sent to a site?

Sure, but you'd need to define how it should work. Camera & mic mute on the web today is controlled by sites, not browsers.

  • Should sites be able to unmute themselves after the user clicks this new "Mute" button?
    • If yes, it might violate many users' expectations of privacy
    • If no, then this new browser-mute control & the site's mute button must BOTH be off or users suffer silence, a usability headache.

In my experience, it's hard enough for users to unmute themselves as it is in a web conference. I would tread carefully altering this UX.

It would also require site support to show the correct in-content mute indicators in these situations, as I fear few sites listen to track.onmute (*) today, as only Safari implements some sort of browser-mute, but it's tied to their limited only-one-tab-can-capture model, which is clunky IMHO.


*) FYI the spec defines track.muted which is "out of control for the application", and is distinct from track.enabled which "is available to the application to control".

Flags: needinfo?(jib)

I'm not saying it would be impossible, but would require a clear model of how it should work, and likely spec support and buy-in from other vendors.

E.g. we might be able to experiment with user-gesture + track.enabled = true to unmute browser-mute, but since browsers aren't allowed to touch the track.enabled state, it would still require site involvement today, e.g. minimally

track.onmute = () => updateMyIndicator(track.enabled = false);
track.onunmute = () => updateMyIndicator(track.enabled = true);
button.onclick = () => updateMyIndicator(track.enabled = !button.checked);

That said, I blame the new UX for suggesting these buttons should mute. We've had behaviorally equivalent indicators for years, and no reported confusion over how they work. They help users locate the tab that's capturing.

I'm going to run jib's concerns by UX. If work proceeds to try to clear up the meaning of the camera / microphone buttons, we'll do that in this bug.

Priority: -- → P3

Please also consider how this will work with multiple tabs using camera and/or microphone at the same time.

The old UX handles this fine (still in toolbar if you're on macOS): You get a dropdown listing all tabs by name that are currently accessing the camera/microphone. You click on the name of the tab of interest and you're taken there to inspect its permissions list or revoke with one click.

Quickly locating the tab(s) using camera/mic is useful, especially with 100s of tabs open.

With the new UX how do I do this (on Windows and Linux)?

Which tab(s) should be muted/unmuted instead? All of them?

That might be a privacy issue. Consider:

  1. I'm in a zoom.us (web-client) room in tab A
  2. I mute camera and microphone using the new browser UX
  3. I open a new tab B and enter a meet.google.com room.
  4. I unmute camera and microphone using the new browser UX.

Does it unmute both zoom.us and meet.google.com? Is that the behavior users would expect? Might some users think they're talking to one group and not realize they're actually talking to two?

Having consulted with UX and Product, we've decided to try to turn the microphone and camera buttons into global mute toggles.

There are a few reasons for this:

  1. The current indicator and behaviour on Linux and Windows is unintuitive. The icons, when clicked, send you to the most recent window and tab that has streams being shared, and then opens the permissions panel just for that tab. This is a really bizarre context transition, to go from global state, to individual tab state. It's also somewhat redundant - we're forwarding the user from one indicator of state, to another indicator of state (albeit one with the ability to revoke the permissions for that state).

  2. We believe that global mute of camera and microphone is just generally more useful as a way for users to have control over what they're sharing over WebRTC.

Consider a use case where I am working from home, and am both in a video conference, and also recording something in a separate tab using the microphone. Suppose one of my children comes in with a question, and I don't want my colleagues nor my recording to hear our conversation. I have a few options: if I'm lucky enough to have a microphone with a hardware mute switch, I could use that. Or I could go into each individual application and mute myself. The first is not guaranteed (I don't think hardware microphone or camera cutoffs are typical), and the second is very awkward. A global mute control like this allows me to stop all audio going out to any and all applications simultaneously, and then resume them when I'm ready to get back to work.

We're hard pressed to think of a similar use case of the current implementation that's as compelling. Sending users into a permission panel to do the revocation of a permission when they want to quickly mute seems like a sub par experience. Unmuting is similarly awkward, as now the user has to go through the flow of re-granting permissions for a particular device. Moreover, revoking the permissions is even known to cause some applications to stop working altogether. Talky.io, for example, breaks if you revoke the camera permission, forcing the user to reload the page to resume a video conference.

We believe we're solving a real problem here by giving users a quick, easy, immediate and smooth way to mute and unmute their cameras and microphones, without having to go through the ins, outs and quirks of whatever web applications they're dealing with. From a privacy and user control perspective, this seems generally like just the right thing to do.

As I've gathered, there are a few concerns here with the idea of a global mute, which I will now attempt to address:

  1. Pre-existing muscle memory, and the loss of finding tabs being shared

There are users that are used to the old pattern of clicking on the indicator to take them to a tab that they're sharing from. The proposed design no longer has a flow for this. We hypothesize that the number of users who have so many tabs that they've lost track of which ones they're sharing streams over, is small. We believe we are serving the interests of more users with the simpler and more intuitive global mutes.

  1. Users being confused by multiple layers of mute

We actually believe this is something most of our users can understand, and won't be confused by. Presuming the indicator hasn't been minimized or moved offscreen by the user, we expect that the "mute" state of the indicator will make it clear when a global mute is applied. We expect that the "always-on-top" nature of the indicator will reinforce the notion that the state being displayed is global, and not tab-specific. The scenario that jib brings up in comment 11, at step 3, we expect the user is aware that they've shared some devices because they will have gone through the permission dialog. We expect the user to understand that the global indicator applies to the new tab as well.

  1. Fingerprinting

jib brings up a really interesting scenario in bug 1654916 comment 9 - the idea that sites across different domains that a user is sharing streams over can collaborate to identify a user by comparing times of mute events firing. There are probably a number of ways of working around this. The one that comes to mind immediately for me, is only immediately firing mute events from the global mute on tabs that are in the foreground. Tabs in the background should then get their mute events fired either after some random amount of time, or perhaps only when the user foregrounds the background tab. That's just off the top of my head - certainly, however, I believe this fingerprinting vector can be noised enough to no longer be an issue.

Having evaluated these concerns, we're still confident that the global mute control is the right path. If there are more concerns that I've failed to mention, please bring them up.

Does this mean we'll remove the macOS indicators in comment 11? Those work great IMHO. Pity we never fixed Linux and Windows to work that way.

I have questions:

  1. Is there a UX mock showing the buttons in muted state? Users will need to decode the four states.
  2. If I mute, and then open a new tab that asks for cam/mic, will the new stream start muted?
  3. If I mute, then close the tab (indicators disappear), then open a new tab that asks for cam/mic, will the new stream start muted (and will the indicators reappear muted)?
  4. If I mute, then the site decides to stop the camera (indicators disappear), and then the site asks for cam/mic anew, will the new stream start muted?

It's a global indicator that trumps site controls, yet whose appearance is controlled by the site?

#4 is a trick question, because that's what Google Meet does to mute camera.

If we go forward with this it will further reveal differences between sites:

  • Most sites, like whereby.com, use track.enabled == false to mute camera and microphone (red→gray Firefox indicator)
  • Google Meet however, uses track.stop() to mute camera—but not mic!—and then calls getUserMedia again to unmute, because that's the only way to remove the camera hardware light on mute in Chrome (we're pushing on Chrome to work like firefox to avoid annoying Firefox permission re-prompts on unmute in Meet).

This difference is observable today (see screenshot): In Meet the global camera indicator goes away on camera mute, whereas the microphone one doesn't go away on mic mute.

Here's #4 again applied to Google Meet:

  1. I'm in Google Meet with camera and microphone shared
  2. I mute camera using the Firefox mute indicator
  3. I mute camera using Meet's mute indicator (causes Firefox indicator to go away)
  4. I unmute camera using Meet mute indicator (causes Firefox indicator to reappear)
  5. Am I muted?

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #13)

Does this mean we'll remove the macOS indicators in comment 11? Those work great IMHO. Pity we never fixed Linux and Windows to work that way.

No, we have no plans to remove the macOS menu bar indicator.

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #14)

  1. Is there a UX mock showing the buttons in muted state? Users will need to decode the four states.

Not yet. shorlander is working on this.

  1. If I mute, and then open a new tab that asks for cam/mic, will the new stream start muted?

Yes.

  1. If I mute, then close the tab (indicators disappear), then open a new tab that asks for cam/mic, will the new stream start muted (and will the indicators reappear muted)?

No. We're going to run with the convention of: if the global indicator closes, then the mute state for all devices resets.

  1. If I mute, then the site decides to stop the camera (indicators disappear), and then the site asks for cam/mic anew, will the new stream start muted?

If the global indicator goes away, then the mute state resets. If the global indicator did not go away (which applies to your Google Meet case), then the mute state will not reset.

It's a global indicator that trumps site controls, yet whose appearance is controlled by the site?

I don't think I fully understand this question. Clearly, the site mute buttons (if they exist) are controlled by the site. The site has no influence over the appearance of the global indicator and the global mute buttons.

#4 is a trick question, because that's what Google Meet does to mute camera.

Here's #4 again applied to Google Meet:

  1. I'm in Google Meet with camera and microphone shared
  2. I mute camera using the Firefox mute indicator
  3. I mute camera using Meet's mute indicator (causes Firefox indicator to go away)
  4. I unmute camera using Meet mute indicator (causes Firefox indicator to reappear)
  5. Am I muted?

Since the microphone was still being shared at step 3, this means the global sharing indicator never went away despite the camera no longer being shared. This means that when the camera was re-shared, the global mute state for the camera was still "muted", and so the camera is muted.

If, however, we add a step 2.5 where the microphone permission is revoked, or modify step 1 so that he microphone is never shared, this will mean that at step 3, muting the camera will cause the global indicator to close. At that point, the global mute state will reset. After re-sharing the camera, the camera will not be muted.

(In reply to Mike Conley (:mconley) (:⚙️) from comment #15)

No, we have no plans to remove the macOS menu bar indicator.

Great! Do we have any plans to build the same great feature for our Windows and Linux users?

If the global indicator goes away, then the mute state resets.

Hmm, so a site with persistent camera/mic permission can circumvent browser mute—unmute itself—just by stopping both tracks and reacquiring them? That seems surprising, and could be a privacy issue.

It's a global indicator that trumps site controls, yet whose appearance is controlled by the site?

I don't think I fully understand this question.

Sorry I meant s/appearance/presence/ - it seems a bit odd, and not clear about which control is in charge.

I worry it's already going to be a challenge to look at the screen and tell whether I'm muted or not.

Like seeing a red and green traffic light at once.

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #16)

Great! Do we have any plans to build the same great feature for our Windows and Linux users?

No plans at this time, but I certainly think it'd be good to be consistent across platforms where possible. So maybe someday?

Hmm, so a site with persistent camera/mic permission can circumvent browser mute—unmute itself—just by stopping both tracks and reacquiring them? That seems surprising, and could be a privacy issue.

Perhaps. They'll still need to go through the device selection / permission flow, so the user will still at least be aware that the devices are being reacquired.

Sorry I meant s/appearance/presence/ - it seems a bit odd, and not clear about which control is in charge.

We suspect that the always-on-top nature of the indicator will make it clear that it is ultimately in charge.

I worry it's already going to be a challenge to look at the screen and tell whether I'm muted or not.

Possibly, but UX doesn't think this will be a problem. I'm still waiting on the spec from shorlander, but will proceed according to plan.

I think it goes without saying that, upon implementing, Product and UX feel like user confusion is certain, we won't ship the design. :)

Perhaps. They'll still need to go through the device selection / permission flow, so the user will still at least be aware that the devices are being reacquired.

Not if they've checked ☑ Remember this decision unfortunately, then there's no prompt, and they'll be unaware.

Firefox is an anomaly in attempting to do device selection in its permission prompt. Instead, the web model has been that sites build device selectors in-content and after-the-fact (go with browser default initially).

The weakness of Firefox's picker is it only appears when permission is needed, and thus can't fully replace this model or in-content device pickers today. The good news is other vendors appear to be coming around to our model, but it will require spec work, which we're pushing (and would love to have some UX resources for).

If the global indicator goes away, then the mute state resets.

Since I didn't expect this answer, I have to ask: What if a site calls getUserMedia a second time to obtain a second stream from A) the same camera, or B) a different camera? Assuming these succeed, will these tracks have to start out muted?

Note this will all be JS observable, since we're required by spec to create all new tracks unmuted and immediately fire the mute event on them.

Depends on: 1652884
Depends on: 1658353

Hey dbodea,

This is the last of the changes to the WebRTC sharing indicator behaviour.

Flags: needinfo?(daniel.bodea)
Attachment #9169433 - Attachment description: Bug 1643027 - [WIP] Add SharedData listeners in WebRTCChild for globally muting the microphone and camera. r?pbz! → Bug 1643027 - Add GlobalMuteListener to WebRTCChild for globally muting the microphone and camera. r?pbz!
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/51c592eb4ebd
Add GlobalMuteListener to WebRTCChild for globally muting the microphone and camera. r=pbz
https://hg.mozilla.org/integration/autoland/rev/aa5d33b0d0b5
Turn the microphone and camera icons in the WebRTC indicator into global mute toggles. r=pbz,fluent-reviewers,flod
https://hg.mozilla.org/integration/autoland/rev/e50501b54674
Add tests for the WebRTC global mute toggles. r=pbz
Blocks: 1661474

Backed out 6 changesets (bug 1643027, bug 1658353) for browser_parsable_css.js failures.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=OLxFQK_jTIenzFDRrv5Apw.0&fromchange=e50501b546744c27d03e013dfc2039dc01a65643&tochange=4933a84ef41e9f6d9b959da1af16008b5843bca4&searchStr=mochitest-browser-chrome

Backout link: https://hg.mozilla.org/integration/autoland/rev/4933a84ef41e9f6d9b959da1af16008b5843bca4

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314189635&repo=autoland&lineNumber=2779

[task 2020-08-27T15:15:34.819Z] 15:15:34     INFO - TEST-START | browser/base/content/test/static/browser_parsable_css.js
[task 2020-08-27T15:15:35.611Z] 15:15:35     INFO - TEST-INFO | started process screenshot
[task 2020-08-27T15:15:35.671Z] 15:15:35     INFO - TEST-INFO | screenshot: exit 0
[task 2020-08-27T15:15:35.672Z] 15:15:35     INFO - <snipped 57 output lines - if you need more context, please use SimpleTest.requestCompleteLog() in your test>
[task 2020-08-27T15:15:35.672Z] 15:15:35     INFO - Buffered messages logged at 15:15:35
[task 2020-08-27T15:15:35.673Z] 15:15:35     INFO - Console message: [JavaScript Warning: "Unknown pseudo-class or pseudo-element ‘-moz-ruby-base-container’.  Ruleset ignored due to bad selector." {file: "resource://gre-resources/ua.css?always-parse-css-0.4985088373806913" line: 124}]
...
...
...
[task 2020-08-27T15:15:35.718Z] 15:15:35     INFO - Console message: [JavaScript Warning: "Expected media feature name but found ‘-moz-windows-default-theme’." {file: "jar:file:///Z:/task_1598537966/build/application/firefox/browser/features/formautofill@mozilla.org.xpi!/chrome/res/autocomplete-item.css?always-parse-css-0.4985088373806913" line: 12}]
[task 2020-08-27T15:15:35.718Z] 15:15:35     INFO - Buffered messages finished
[task 2020-08-27T15:15:35.718Z] 15:15:35     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_parsable_css.js | custom property `--device-button-hover-background-color` is not referenced - 
[task 2020-08-27T15:15:35.719Z] 15:15:35     INFO - Stack trace:
[task 2020-08-27T15:15:35.719Z] 15:15:35     INFO - chrome://mochikit/content/browser-test.js:test_ok:1304
[task 2020-08-27T15:15:35.719Z] 15:15:35     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_parsable_css.js:checkAllTheCSS:460
[task 2020-08-27T15:15:35.719Z] 15:15:35     INFO - Not taking screenshot here: see the one that was previously logged
[task 2020-08-27T15:15:35.720Z] 15:15:35     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_parsable_css.js | custom property `--device-button-active-background-color` is not referenced - 
[task 2020-08-27T15:15:35.720Z] 15:15:35     INFO - Stack trace:
[task 2020-08-27T15:15:35.720Z] 15:15:35     INFO - chrome://mochikit/content/browser-test.js:test_ok:1304
[task 2020-08-27T15:15:35.720Z] 15:15:35     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_parsable_css.js:checkAllTheCSS:460
[task 2020-08-27T15:15:35.721Z] 15:15:35     INFO - Ignored error "Unknown pseudo-class or pseudo-element ‘-moz-has-dir-attr’.  Ruleset ignored due to bad selector." on resource://gre-resources/html.css because of whitelist item {"sourceName":"/\\b(contenteditable|EditorOverride|svg|forms|html|mathml|ua|pluginproblem)\\.css$/i","errorMessage":"/Unknown pseudo-class.*-moz-/i","isFromDevTools":false,"used":true}
...
...
...
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - Ignored error "Expected media feature name but found ‘-moz-windows-default-theme’." on jar:file:///Z:/task_1598537966/build/application/firefox/browser/features/formautofill@mozilla.org.xpi!/chrome/res/autocomplete-item.css because of whitelist item {"sourceName":"/\\b(autocomplete-item|svg|ua)\\.css$/","errorMessage":"/Expected media feature name but found \\u2018-moz.*/i","isFromDevTools":false,"used":true}
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - TEST-PASS | browser/base/content/test/static/browser_parsable_css.js | All the styles (260) loaded without errors. - 
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - Leaving test bound checkAllTheCSS
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - GECKO(5732) | MEMORY STAT | vsize 2104728MB | vsizeMaxContiguous 67071575MB | residentFast 818MB | heapAllocated 613MB
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - TEST-OK | browser/base/content/test/static/browser_parsable_css.js | took 867ms
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - checking window state
[task 2020-08-27T15:15:35.776Z] 15:15:35     INFO - TEST-START | browser/base/content/test/static/browser_parsable_script.js
[task 2020-08-27T15:15:54.947Z] 15:15:54     INFO - GECKO(5732) | MEMORY STAT | vsize 2104669MB | vsizeMaxContiguous 67071575MB | residentFast 670MB | heapAllocated 441MB
[task 2020-08-27T15:15:54.947Z] 15:15:54     INFO - TEST-OK | browser/base/content/test/static/browser_parsable_script.js | took 19236ms
Flags: needinfo?(mconley)
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/daa5e1839f76
Add GlobalMuteListener to WebRTCChild for globally muting the microphone and camera. r=pbz
https://hg.mozilla.org/integration/autoland/rev/13504af843f3
Turn the microphone and camera icons in the WebRTC indicator into global mute toggles. r=pbz,fluent-reviewers,flod
https://hg.mozilla.org/integration/autoland/rev/35ffafc46c68
Add tests for the WebRTC global mute toggles. r=pbz
Flags: needinfo?(mconley)
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 82 Branch

I have tested this feature with a few apps and it seems to be working as intended, but I would like to leave the validation of this bug for a full testing session of the feature to make sure all considered WebRTC web-apps behave accordingly, considering the global level of camera and mic muting implemented.

That being said, I will leave my NI on this bug until it is tested completely.

The behavior of the mute toggles (that have to be manually activated) is properly working. I deem this bug verified. Thank you.

Status: RESOLVED → VERIFIED
Flags: needinfo?(daniel.bodea)

This bug is fixed.
However the bug number still exists in toolkit/components/featuregates/Features.toml.

(In reply to Takanori MATSUURA from comment #31)

This bug is fixed.
However the bug number still exists in toolkit/components/featuregates/Features.toml.

That's because the feature is disabled by default. It's controlled by the pref here: https://searchfox.org/mozilla-central/rev/684aefab97202faa982dd898f1d99208a46f12b0/browser/app/profile/firefox.js#1969

You need to log in before you can comment on or make changes to this bug.