877248 - Preview Surface for Camera is visible in upper left corner

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Gian-Carlo Pascutto [:gcp]]

Android requires a preview surface to connect the camera to. Pre-Android 4.0, some tricks must be used which make the surface invisible on most phones.

From my understanding it's impossible to make this surface go away on all phones, but I'd like to to be hidden on as much as possible.

Currently, it's visible as a small white squire on my Android 3.2 Galaxy 10.1. It's not visible on the Nexus 4 and Galaxy S3 (even though we don't use the Android4+ method to ensure that).

I'd like it if QA could report their findings per phone here.

Comment 2

[Aaron Train [:aaronmt]]

Of the devices I have tested most of gUM on thus far here is what I have seen:

Visible surface:

* Asus Transformer Prime TF201 (Android 4.1)
* Asus Nexus 7 (Android 4.2)

No visible surface:

* Alcatel One Touch 8008X (Android 4.1)
* HTC One (Android 4.1)
* HTC Nexus One (Android 2.3)
* Samsung Galaxy SII (Android 4.0 & Android 4.1)
* Samsung Galaxy SIII (Android 4.1)
* Samsung Galaxy Note - (Android 4.1)
* Samsung Galaxy Note II - (Android 4.1)
* Samsung Nexus S (Android 4.1)
* Samsung Galaxy Nexus (Android 4.2)
* LG Nexus 4 (Android 4.2)

Interestingly only the two tablets I have looked at so far exhibit the upper white dot (see https://bugzilla.mozilla.org/attachment.cgi?id=749445). I'll be taking a look at a few more next week.

Comment 3

[Gian-Carlo Pascutto [:gcp]]

This is very useful information already. We should do the "guaranteed no visible square" thing for Android 4.x as that will at least fix your two bad devices.

Comment 4

[Aaron Train [:aaronmt]]

Attachment: Samsung Galaxy SIV (Screenshot)

Surface preview

* Samsung Galaxy SIV (Android 4.2)

Comment 5

[Aaron Train [:aaronmt]]

Attachment: LG Nexus 4 (Screenshot)

I retract on the Nexus 4 (4.2.2), I do see the surface preview image only when I animate open the tab tray for whatever reason

Comment 6

[Aaron Train [:aaronmt]]

Attachment: LG Nexus 4 (Screenshot)

Actually this is obvious to see on a new private tab

Comment 7

[Jason Smith [:jsmith]]

After seeing this myself, I'll retract my original non-blocking comment. I didn't realize the little white blob in the corner was the camera until I looked closer. That's probably worth blocking on.

Comment 8

[Soledad Penades [:sole] [:spenades]]

I got this yesterday too just after updating to the latest Nightly build. I hadn't seen this before--or hadn't noticed it before!

Using an LG Nexus 4, Android 4.2.2

I had to "downgrade" to a build from past week (7th June) because I couldn't get any WebRTC camera stuff to work (getting NS_FAILURE or similar).

Comment 9

[Gian-Carlo Pascutto [:gcp]]

>After seeing this myself, I'll retract my original non-blocking comment.

As explained in the initial comment, this bug is not fixable in the general case (for Android < 4.0), so I don't think we can block on it.

>I had to "downgrade" to a build from past week (7th June) because I couldn't get 
>any WebRTC camera stuff to work (getting NS_FAILURE or similar).

Which bug is that filed as?

Comment 10

[Jason Smith [:jsmith]]

(In reply to Gian-Carlo Pascutto (:gcp) from comment #9)
> >After seeing this myself, I'll retract my original non-blocking comment.
> 
> As explained in the initial comment, this bug is not fixable in the general
> case (for Android < 4.0), so I don't think we can block on it.

Why can't this be fixed on Android < 4.0? We can't even introduce a workaround to hide the preview surface?

Comment 11

[Soledad Penades [:sole] [:spenades]]

(In reply to Gian-Carlo Pascutto (:gcp) from comment #9)
> 
> >I had to "downgrade" to a build from past week (7th June) because I couldn't get 
> >any WebRTC camera stuff to work (getting NS_FAILURE or similar).
> 
> Which bug is that filed as?

It was an strange coincidence, but what seemed to be WebRTC's fault was actually WebGL. The bug is already fixed resolved :-)
https://bugzilla.mozilla.org/show_bug.cgi?id=881681

Comment 12

[Soledad Penades [:sole] [:spenades]]

(In reply to Jason Smith [:jsmith] from comment #10)
> (In reply to Gian-Carlo Pascutto (:gcp) from comment #9)
> > >After seeing this myself, I'll retract my original non-blocking comment.
> > 
> > As explained in the initial comment, this bug is not fixable in the general
> > case (for Android < 4.0), so I don't think we can block on it.
> 
> Why can't this be fixed on Android < 4.0? We can't even introduce a
> workaround to hide the preview surface?

I haven't looked into the implementation details, but I can guesstimate why, as I used to fight over this exact issue when I did Android programming. You need to display the camera preview on a SurfaceView on Android < 4.0. This SurfaceView area "cuts" through the view and is always on top. The only solution is to hide it by making it small enough. But not too much or some phones fail :-P

Android 4.0 introduced a new class that CAN be hidden.

Comment 13

[Jason Smith [:jsmith]]

(In reply to Soledad Penades [:sole] [:spenades] from comment #12)
> (In reply to Jason Smith [:jsmith] from comment #10)
> > (In reply to Gian-Carlo Pascutto (:gcp) from comment #9)
> > > >After seeing this myself, I'll retract my original non-blocking comment.
> > > 
> > > As explained in the initial comment, this bug is not fixable in the general
> > > case (for Android < 4.0), so I don't think we can block on it.
> > 
> > Why can't this be fixed on Android < 4.0? We can't even introduce a
> > workaround to hide the preview surface?
> 
> I haven't looked into the implementation details, but I can guesstimate why,
> as I used to fight over this exact issue when I did Android programming. You
> need to display the camera preview on a SurfaceView on Android < 4.0. This
> SurfaceView area "cuts" through the view and is always on top. The only
> solution is to hide it by making it small enough. But not too much or some
> phones fail :-P

Hmm...okay. If we have this restriction, then maybe UX can work some magic to reduce the impact of this bug if we're required to show this.

Ian - Do you have any ideas from a UX perspective on what we can do here to reduce the impact of this limitation on Android < 4.0?

> 
> Android 4.0 introduced a new class that CAN be hidden.

Okay, so we can fix this on Android 4.0 then.

Comment 14

[Ian Barlow (:ibarlow)]

(In reply to Jason Smith [:jsmith] from comment #13)

> Hmm...okay. If we have this restriction, then maybe UX can work some magic
> to reduce the impact of this bug if we're required to show this.
> 
> Ian - Do you have any ideas from a UX perspective on what we can do here to
> reduce the impact of this limitation on Android < 4.0?


Heh, "UX magic". My first suggestion would be to keep looking for a way to hide that tile ;)

Let me make sure I understand this correctly: 
In order for the camera to work, we have to show a small video preview on older devices? 

If that is the case then I have other questions:
* Is this true even now that we have icons in the notification bar indicating the state of a user's camera or microphone?
* What about the actual video preview in web content? How does that relate to this little thing in the top corner?
* Obvious question, but, what do other mobile browsers do on Android? 
* Lastly, if there is simply no way to hide that tile, can we maybe not put it over top of UI chrome, and tuck it in the bottom right corner of the web content area?

Comment 15

[Gian-Carlo Pascutto [:gcp]]

Comment 12 nails it, see also my blog post from a while ago: http://www.morbo.org/2013/02/why-twitter-pictures-are-sideways.html

>* Is this true even now that we have icons in the notification bar indicating the state of a user's camera or microphone?

Yes, it's completely unrelated. It's a technical limitation of pre-ICS Android.

>* What about the actual video preview in web content? How does that relate to this little thing in the top corner?

No relation at all. The webpages can do anything: show a preview, can show the remote video, both or neither of those things. But the local video *must* be previewed somewhere on the screen, and I don't really see a way to reuse the preview (which would be in a <video> element!) that the webpage *might* have as we have no idea how or where that will be.

Basically we're stuck with having to put that preview somewhere visible on screen. But only on select pre-ICS devices where none of the workarounds work.

>* Obvious question, but, what do other mobile browsers do on Android? 

Not work. [1]

>* Lastly, if there is simply no way to hide that tile, can we maybe not put it over top of UI chrome, and tuck it in the bottom right corner of the web content area?

Yeah, that seems like the most sensible thing to do.

[1] Not support WebRTC on pre-ICS Android, more specifically.

Comment 16

[Ian Barlow (:ibarlow)]

> >* Lastly, if there is simply no way to hide that tile, can we maybe not put it over top of UI chrome, and tuck it in the bottom right corner of the web content area?
> 
> Yeah, that seems like the most sensible thing to do.

Can we make it as small as possible then? Would a 1dp x 1dp square still allow it to function?

Comment 17

[Gian-Carlo Pascutto [:gcp]]

>Can we make it as small as possible then? Would a 1dp x 1dp square still allow it to function?

On some devices yes, on some devices, no. QA will have to test what works where.

Comment 18

[Lai Chen Haw]

Note II with Nightly 24.0a1(2013-06-21), I can see the small region appear at top left corner. Hope this help.

Comment 19

[Hong SeGi]

* Nexus 7 (Android 4.2.2)
* Galaxy Nexus (Android 4.2.2)

Both of devices with 2013-06-21.

In case of Galaxy Nexus, to reproduce a problem, you should zoom page and make title bar visible.

https://www.dropbox.com/sc/v15idgpk75ezsp0/MadZThpJOR

Comment 20

[Gian-Carlo Pascutto [:gcp]]

Attachment: Patch 1. Make Surface invisible where possible

1) Despite claims that you can make the View of a SurfaceTexture/TextureView invisible and have the camera work, this didn't seem to work on my test devices. However what does seem to work is to set the alpha channel of the texture to transparent. (On HoneyComb this takes effect after a delay, or at least it looks like this, very strange). I'm leaving in the code to use TextureView instead of SurfaceView where supported, because I presume that does actually work faster/better where it exists.

2) You can't have a class that implements an interface that belongs to a higher SDK than the one on the device. The classloader will fail. This is worked around with making the SurfaceTextureListener an inner class that is only instantiated if you're on the right device SDK level.

3) Modifications to the layout so the Surface, if visible, should be near on the lower right instead.

QA should prolly play with the try build before we land this:
https://tbpl.mozilla.org/?tree=Try&rev=42654a5381a2

Comment 21

[Adrian Tamas (:AdrianT)]

Tested the try-build on Acer Iconia A500 (Android 3.2), Motorola RAZR (Android 2.3.3) and Samsung Galaxy R (Android 2.3.4) and the preview surface is not visible when testing on http://mozilla.github.io/webrtc-landing/gum_test.html. On the latest Nightly all these devices display the surface view so this patch fixes the issue.

Comment 22

[Jason Smith [:jsmith]]

(In reply to Adrian Tamas from comment #21)
> Tested the try-build on Acer Iconia A500 (Android 3.2), Motorola RAZR
> (Android 2.3.3) and Samsung Galaxy R (Android 2.3.4) and the preview surface
> is not visible when testing on
> http://mozilla.github.io/webrtc-landing/gum_test.html. On the latest Nightly
> all these devices display the surface view so this patch fixes the issue.

Can you try an Android 4.0 or higher device as well?

Comment 23

[Gian-Carlo Pascutto [:gcp]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4f24e84e1da8

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/4f24e84e1da8

Comment 25

[Jason Smith [:jsmith]]

Note - this needs to be uplifted to Aurora.

Comment 26

[Gian-Carlo Pascutto [:gcp]]

Comment on attachment 768387 [details] [diff] [review]
Patch 1. Make Surface invisible where possible

[Approval Request Comment]
Bug caused by (feature/regressing bug #): WebRTC support
User impact if declined: White square visible over browser chrome when getUserMedia is used.
Testing completed (on m-c, etc.): Landed on m-c few days ago
Risk to taking this patch (and alternatives if risky): White square elsewhere, or other general browser chrome/layout breakage. 
String or IDL/UUID changes made by this patch: NA

Comment 27

[Gian-Carlo Pascutto [:gcp]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/9666173e5d61