No notification visible indicator implemented for when getUserMedia audio is active

RESOLVED FIXED in 1.2 C4(Nov8)

Status

defect
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: jsmith, Assigned: gasolin)

Tracking

({late-l10n})

unspecified
1.2 C4(Nov8)
ARM
Gonk (Firefox OS)
Dependency tree / graph

Firefox Tracking Flags

(blocking-b2g:koi+, b2g-v1.2 verified)

Details

(Whiteboard: [ft:system-platform])

Attachments

(9 attachments)

Build: Master 9/17/2013
Device: Inari

STR

1. Go to http://mozilla.github.io/webrtc-landing/gum_test.html
2. Go to audio
3. Grant permissions
4. Pull down the notification bar

Expected

The visible indicator should be present in the notification bar indicating that the mic is active at a target web origin.

Actual

The visible indicator isn't present in the notification bar.
Blocks: 894848
blocking-b2g: --- → koi?
I think audio capture would trigger recording state changes -> then mozChromeEvent with event.detail.type = recording-status would be sent to system app.
SC^? However I recalled that I had seen the red icon on the statusbar during oslo workweek when we tested gum...
There is a red dot on notification bar while recording start already. What @jsmith means is that we should have an additional indicator on the notification window like the snapshot from UI spec I attached.
OK, I saw it in the document :/
Questions:
Could we know the recording time past in the chrome event?
Could we know who(page) is doing recording in the chrome event?
BTW, I tried to trace 'recording-device-events' in gecko, and it seems we are using this event for both audio and video recording. We need to distinguish.
http://mxr.mozilla.org/mozilla-central/search?string=recording-device-events

Looks like we need to change lots of codes to implement "audio capture notification" + "video capture notification" because the original event only has RecordingStatus info.
Indeed. From the UI spec we'll need add "type" and "origin" information in the mozChromeEvent. As for the time information, it should be the time that Gaia receive the mozChromeEvent. I think Gaia can get current time directly in javascript without read from mozChromeEvent.
(In reply to Shih-Chiang Chien [:schien] from comment #6)
> Indeed. From the UI spec we'll need add "type" and "origin" information in
> the mozChromeEvent. As for the time information, it should be the time that
> Gaia receive the mozChromeEvent. I think Gaia can get current time directly
> in javascript without read from mozChromeEvent.

Uh, if I understand correctly, we need to know "recording time past". How do we know from javascript?
(In reply to Shih-Chiang Chien [:schien] from comment #6)
> Indeed. From the UI spec we'll need add "type" and "origin" information in
> the mozChromeEvent. As for the time information, it should be the time that
> Gaia receive the mozChromeEvent. I think Gaia can get current time directly
> in javascript without read from mozChromeEvent.

BTW We should use 'manifestURL' + 'pageURL' to identify a page, not origin.
Note: the desktop UI uses the recording-device-events notification as an indicator it needs to check if any of the active captures have been added or gone away:

function updateIndicators() {
  webrtcUI.showGlobalIndicator =
    MediaManagerService.activeMediaCaptureWindows.Count() > 0;

  let e = Services.wm.getEnumerator("navigator:browser");
  while (e.hasMoreElements())
    e.getNext().WebrtcIndicator.updateButton();

  for (let {browser: browser} of webrtcUI.activeStreams)
    showBrowserSpecificIndicator(browser);
}

showBrowserSpecificIndicator queries the capture state with each active capturing contentWindow for audio and video capture.  This includes any embedded iframes using webrtc, etc, and lets us have a list of active what that window is capturing (which you may not need).

So I don't think any API changes are needed to get the info the UI needs.

See browser/modules/webrtcUI.jsm
Thanks for the info.
It looks like MediaManagerService.activeMediaCaptureWindows has everything we need.

Maybe we could parse these info in shell.js and pass it to system?
Blocks: b2g-getusermedia
No longer blocks: 894848
Assignee: nobody → alive
In shell.js, I found MediaManagerService.activeMediaCaptureWindows.Count() is 0 even when audio capture is running :/
Component: Gaia::System → General
activeMediaCaptureWindows should work; it's used on desktop per above to enable/disable the active-capture-window dropdown

Perhaps it's something in shell.js?  Perhaps it's running in the wrong process?  (I know nothing about shell.js on B2G)
(In reply to Randell Jesup [:jesup] from comment #12)
> activeMediaCaptureWindows should work; it's used on desktop per above to
> enable/disable the active-capture-window dropdown
> 
> Perhaps it's something in shell.js?  Perhaps it's running in the wrong
> process?  (I know nothing about shell.js on B2G)

Yes, because shell.js running in chrome process but the app having media manager service running in content process.
We need to send the recording-device-events appending with mActiveWindow, not only recording status.

http://mxr.mozilla.org/mozilla-central/source/dom/ipc/PContent.ipdl#461
I think I need at least manifestURL + pageURL to identify the app/page.
manifestURL is browserElement.getAttribute('mozapp').
needinfo : Ivan here to help the decision on the bug.

From  the read of it and my discussion with :jsmith it looks like a feature miss, but we may have privacy concern's if this is left um-implemented.

So, Ivan how do we want to move forward wrt 1.2 here ?
Flags: needinfo?(itsay)
Mentioned this in person to rel management, but main sticking point for blocking on this is that a user needs to have a clear understanding of what device is active. Right now, the recording indicator is used interchangeably with getUserMedia and the camera app. That means that a user right now can distinguish that recording is active, but not understand which device (microphone or camera) is doing the recording. That's a privacy concern if the user cannot distinguish that.
Extract the gecko part to bug 926289.
Change component of this bug to gaia::system
Component: General → Gaia::System
Patryk, who could provide the notification recording icon asset?
Flags: needinfo?(padamczyk)
Flags: needinfo?(itsay)
(In reply to bhavana bajaj [:bajaj] from comment #15)
> needinfo : Ivan here to help the decision on the bug.
> 
> From  the read of it and my discussion with :jsmith it looks like a feature
> miss, but we may have privacy concern's if this is left um-implemented.
> 
> So, Ivan how do we want to move forward wrt 1.2 here ?

Hi Bhavana,

This one should not be the feature miss but a privacy issue on gUM. While planning this Microphone API gUM, I and CJ committed to finish A3 of UX design on gUM (please see attached image). However, in this bug, it require to finish all up to A5. After discussing it with Alive, I can still plus this one and we will try to finish it by Oct. 28.
blocking-b2g: koi? → koi+
Add keyword for tracking
Whiteboard: [ft:system-platform]
(In reply to Ivan Tsay (:ITsay) from comment #20)
> (In reply to bhavana bajaj [:bajaj] from comment #15)
> > needinfo : Ivan here to help the decision on the bug.
> > 
> > From  the read of it and my discussion with :jsmith it looks like a feature
> > miss, but we may have privacy concern's if this is left um-implemented.
> > 
> > So, Ivan how do we want to move forward wrt 1.2 here ?
> 
> Hi Bhavana,
> 
> This one should not be the feature miss but a privacy issue on gUM. While
> planning this Microphone API gUM, I and CJ committed to finish A3 of UX
> design on gUM (please see attached image). However, in this bug, it require
> to finish all up to A5. After discussing it with Alive, I can still plus
> this one and we will try to finish it by Oct. 28.

It's technically a feature miss if the full feature didn't land in time for feature complete. This was part of the core spec for the feature which should have been implemented before feature complete. This is particularly important because implementations like these will require strings, so the later this is implemented, the more localization risk becomes apparent.
Alive, we added a recording tone with bug, please reference it.
https://bugzilla.mozilla.org/show_bug.cgi?id=896874
Flags: needinfo?(padamczyk)
Patryk, we need the real image of 'red dot'(?) shown in spec (Left icon for Notification message)
https://bug917367.bugzilla.mozilla.org/attachment.cgi?id=807149

Can you help provide the icon assets?
Flags: needinfo?(padamczyk)
taken
Assignee: alive → gasolin
Peter La should be able to provide the icon.
Flags: needinfo?(padamczyk) → needinfo?(pla)
Currently gUM blocked by bug 926746. it means we can't test on real device yet.
But I'd start from UI-related work first.
Depends on: 926746
Target Milestone: --- → 1.2 C4(Nov8)
WIP: take notification style and reuse gUM permission list icons
gUM permission list icons are 64x64, still expect replaced by smaller icons
WIP

alive, could you kindly check if it's the expect way to implement this feature?
will add unittest if its in right path.
Attachment #821444 - Flags: feedback?(alive)
Comment on attachment 821444 [details] [review]
pull request redirect to github

See github.
Attachment #821444 - Flags: feedback?(alive) → feedback+
Peter, please provide Video(camera)/Media(recorder) icons as well, so we not need to do it again in 1.3.
PR is ready but need add unittest, we'll add 3 strings into system locales, add l10n team to check early

microphone-is-on=Microphone is on
camera-is-on=Camera is on
media-is-on=Camera and Microphone are on
Flags: needinfo?(l10n)
I would expect that in particular the media-is-on string exceeds a single line, can't tell from the screenshots if that works, though.

Note, needinfo on me for string changes isn't necessary. You need to coordinate with release management on when you need to be done. Our team is focusing on getting the strings that are landed out to localizers.
Flags: needinfo?(l10n)
Keywords: late-l10n
thanks, BTW I've checked media-is-on string just fit the single line.
English:
Camera and Microphone are on

German:
Kamera und Mikrofon eingeschaltet

You should expect that localizations are longer than English.
French just came in on irc:

La caméra et le microphone sont en marche
patryk, would you like to suggest some shorter 'media-is-on' string? thanks.

While following the gUM UI spec, we also find many inconsistency places such as the gUM icons & strings are vary, and some dialogs still have 'remember' toggle button, which should not be shown there.
Hope we could get an more accurate version.
Flags: needinfo?(padamczyk)
I have a few suggestions.

1. Microphone seems long winded, its often abbreviated to "MIC", so it could read like this:
- Camera and Mic are on

2. I would reduce the words and emphasis the ON / OFF
- Camera & Mic: ON

Fred, who wrote the gUM spec? Can you send it over and we can scrub the inconsistencies.
Flags: needinfo?(padamczyk)
(In reply to Patryk Adamczyk [:patryk] UX from comment #39)
> I have a few suggestions.
> 
> 1. Microphone seems long winded, its often abbreviated to "MIC", so it could
> read like this:
> - Camera and Mic are on
> 
> 2. I would reduce the words and emphasis the ON / OFF
> - Camera & Mic: ON
> 
> Fred, who wrote the gUM spec? Can you send it over and we can scrub the
> inconsistencies.

Point of clarification - you mean the W3C spec jesup cites in comment 40 or the UX spec?
Patryk, 

I got the UI spec named gUM/WebRTC permissions authored by Esther Leong & Jacqueline Savory, last update date is 8/12/2013
Comment on attachment 821444 [details] [review]
pull request redirect to github

added the unittest and passed travis test

though still need gecko bug resolved to test on real device, and need update with new images
Attachment #821444 - Flags: review?(alive)
I can provide an image that works around the bug 926746 for your to test.
Thanks SC, I've test in real device and works well. (with bug 919927 WIP in master)
Posted image ScreenMockup.png
Hi Fred,

I'm proposing that we use a red dot to cover every case because it's visually stronger and more attention grabbing than using mic and video camera iconography.

The fact that we have l10n text that says 'Microphone is On' or 'Video Camera is On' means we don't need to explicitly show it in the icon.  The combination of the text and red dot is strong.

Attached is a screenshot mocking this up.
Flags: needinfo?(pla)
Here is the icon asset for the red dot.
Hi peter, thanks for provide the icon. 
I think we need 1.5x, 2x icons as well.

Is there any difference between mic/camera/both icons? Or all of them use the same one?
Flags: needinfo?(pla)
(In reply to Peter La from comment #46)
> Created attachment 825928 [details]
> ScreenMockup.png
> 
> Hi Fred,
> 
> I'm proposing that we use a red dot to cover every case because it's
> visually stronger and more attention grabbing than using mic and video
> camera iconography.
> 
> The fact that we have l10n text that says 'Microphone is On' or 'Video
> Camera is On' means we don't need to explicitly show it in the icon.  The
> combination of the text and red dot is strong.
> 
> Attached is a screenshot mocking this up.

No that's not right - that doesn't meet the privacy requirements of the feature. There's a critical need that the user has to have always a clear indication of what recording is active - camera, mic, or both. That means being as explicit as possible across the UX as much as possible. Hiding information anywhere means we risk reducing user knowledge of what's actually active, so we risk cases where the user might think say, the camera is active when the mic is actually active.

We already know that the explicit icon usage in notification indicators worked successfully on Firefox for Android, so I don't really think what's proposed here is the right direction to go to protect against user privacy.

needinfo on Paul for his opinion here as well from the security perspective
Flags: needinfo?(ptheriault)
Attachment #821444 - Flags: review?(alive) → review+
> needinfo on Paul for his opinion here as well from the security perspective

Hmmm I don't know if I feel strongly either way, sorry!  On one hand I sort of agree with the red dot being visually stronger, but then is this at the expense of the attention to other items ? Wont users just ignore any icons that are not red?

Jason's point about losing information seems valid too though, and it seems like a bad idea to have vastly different iconography on Firefox OS vs Firefox Mobile vs Firefox Desktop.

However given where we are/were with audio notification, from my perspective I could live with just the red dot for now (to me the behavior of when and where it is shown is much more important than the actual icon), but I would like us to revisit our approach to security UI as part of Haida effort so that we get some kind of cohesive approach. This is especially in the case of persistent notifications for active permissions, which will only become more important as we expose more sensitive APIs.
Flags: needinfo?(ptheriault)
Posted file Icon Concepts
Hi Jason/Paul,

Thanks for sharing that information.  I've attached 3 icon concepts that show more explicitly each case.  I'd love to get your feedback before we finalize tomorrow.

Fred, I will be providing all of the icons once we finalize, and yes, they will be three distinct icons.
Flags: needinfo?(pla)
(In reply to Peter La from comment #51)
> Created attachment 827174 [details]
> Icon Concepts
> 
> Hi Jason/Paul,
> 
> Thanks for sharing that information.  I've attached 3 icon concepts that
> show more explicitly each case.  I'd love to get your feedback before we
> finalize tomorrow.

I like the direction of the three concepts. I like the red icon with white mic/camera icons the best, as it seems to keep the red icon concept for recording & is readable to provide indication that the camera, mic, or both are active.
(In reply to Jason Smith [:jsmith] from comment #52)
> (In reply to Peter La from comment #51)
> > Created attachment 827174 [details]
> > Icon Concepts
> > 
> > Hi Jason/Paul,
> > 
> > Thanks for sharing that information.  I've attached 3 icon concepts that
> > show more explicitly each case.  I'd love to get your feedback before we
> > finalize tomorrow.
> 
> I like the direction of the three concepts. I like the red icon with white
> mic/camera icons the best, as it seems to keep the red icon concept for
> recording & is readable to provide indication that the camera, mic, or both
> are active.

Great, that's our consensus as well.  The red circle + white icon provides the clearest read.

Fred, Amy Lee will be exporting and attaching the icons @1x and @1.5x size to this bug shortly.
Hi Fred, 

Attached are the icons in @1 and @1.5 scale. Let me know if you need anything else.
merged to gaia-master https://github.com/mozilla-b2g/gaia/commit/00d6e5be8db6851dd202598eddd3d09017e29f46

thanks!
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Amy, thanks for help. We may need a separate permission prompt window icons to present video&Mic case.

currently we use the same icon as video

https://github.com/mozilla-b2g/gaia/blob/master/apps/system/style/permission_manager/images/PermissionsDialogIcons_VideoRecorder.png

please attach the icon in bug 935342
Blocks: 935342
Keywords: verifyme
QA Contact: jsmith
Uplifted 00d6e5be8db6851dd202598eddd3d09017e29f46 to:
v1.2: e18f19c51576a2876fae22be2e442bddac743caa
Comment on attachment 821444 [details] [review]
pull request redirect to github

This really needed a UI review before this landed. I'm adding the ui review flag for Jacqueline to take a look at this to see if what landed aligns with the gUM UX spec.
Attachment #821444 - Flags: ui-review?(jsavory)
Just finished testing this - see the dependencies of bug 918876 recently filed for bugs that were found while testing this. The only bug that was found during this testing that likely is going to block 1.2 is bug 940045.
Jason, feel free to cc me if it's a gaia related confidential bug
(In reply to Fred Lin [:gasolin] from comment #60)
> Jason, feel free to cc me if it's a gaia related confidential bug

I think it's gecko related, but if it ends up becoming gaia related, I'll let you know.
Comment on attachment 821444 [details] [review]
pull request redirect to github

I actually ended up doing a QA pass that looked at some of the UX implications, so I don't think we need to wait on a review at this point. But feel free to look at this point after the fact if you like.
Attachment #821444 - Flags: ui-review?(jsavory)
Duplicate of this bug: 1178287
You need to log in before you can comment on or make changes to this bug.