880780 - Camera - viewfinder is corrupted, mostly green, with random noise

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Mike Habicher [:mikeh] (high bugzilla latency)]

Attachment: Screenshot of camera viewfinder

Observed on Unagi DEBUG build with:
- gecko: m-c:a47f4e36197f
- gaia: v1-train:b5937ae0d91682e5d5fc89591765813fff72c8dc
+ patch from bug 879478 (fixes media stream brokenness)
+ patch (either) from bug 880268 (fixes assert in BytesPerPixelForPixelFormat())

STR:
0. BRANCH=v1-train ./config.sh unagi && B2G_DEBUG=1 ./build.sh && ./flash.sh
1. open the camera app

Expected result: viewfinder works properly

Observed result: viewfinder display is corrupted as in the attached image

An interesting side-effect of capturing the attached screenshot (hold down the power button and press the Home button) is that a correct image appears in the viewfinder very briefly, before going back to the greenly-goodness.

Comment 1

[Mike Habicher [:mikeh] (high bugzilla latency)]

bjacob, sotaro suggested I ping you about this one--that maybe the recent graphics refactoring could have regressed this.

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

Can this be observed only by mixing m-c gecko with v1 gaia? If yes, should we care? m-c gecko + v1 gaia has been severely broken for a long time, no?

Comment 3

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to Benoit Jacob [:bjacob] from comment #2)
>
> Can this be observed only by mixing m-c gecko with v1 gaia? If yes, should
> we care? m-c gecko + v1 gaia has been severely broken for a long time, no?

Sorry about that; too many branchen to juggle these days. Let me retry that with master gaia.

Comment 4

[Mike Habicher [:mikeh] (high bugzilla latency)]

Observed on Unagi DEBUG build with:
- gecko: m-c:a47f4e36197f
- gaia: master:270a5943fffc884e61bccd61209eeca61a63f9b3
+ patch from bug 879478 (fixes media stream brokenness)
+ patch (either) from bug 880268 (fixes assert in BytesPerPixelForPixelFormat())

With this build, the above symptoms are still visible.

I also noticed that pressing and holding the Home button to bring up the task manager shows a thumbnail of the camera app with a correct viewfinder image.

Comment 5

[Mike Habicher [:mikeh] (high bugzilla latency)]

Attachment: Screenshot of camera viewfinder in task manager

Comment 6

[Benoit Jacob [:bjacob] (mostly away)]

OK, that the image is correct in task manager further points to a compositor bug.

I have no idea what regressed it though, I can only point to some ideas:
 - you could run a debug build with MOZ_GL_DEBUG_ABORT_ON_ERROR=1 in your environment to see if there is some OpenGL error along the way.
 - you could check if backing out the patch in bug 875211 makes in a difference: it is the only major change in the b2g compositing path that landed recently, that could cause this kind of regression, that I could think of.

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

Regression range would be awesome.

Comment 8

[Maire Reavy [:mreavy]]

WebRTC is having its work week this week.  We are hoping to make good progress getting WebRTC working on B2G.  This bug is a blocker for our B2G work.  Is it possible for someone to look at this ASAP?  

We're trying to help identify a regression range, but we're hoping someone closer to the compositor code can take the lead.

Thanks in advance for any help prioritizing and fixing this ASAP.

Comment 9

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to Maire Reavy [:mreavy] from comment #8)
> 
> Thanks in advance for any help prioritizing and fixing this ASAP.

Maire, can you try the patch in bug 879478 comment 28?

Comment 10

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to Mike Habicher [:mikeh] from comment #9)
> 
> Maire, can you try the patch in bug 879478 comment 28?

I can save you the time--I just tried this patch and it doesn't fix the problem.

Comment 11

[Mike Habicher [:mikeh] (high bugzilla latency)]

Another data point: taking a picture causes the viewfinder to flash briefly with a correct image as well.

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

Because I was confused for a bit, the relevant branch here is mozilla-central aka master, where webrtc goes to.

Comment 13

[Sotaro Ikeda [:sotaro]]

It seem like it is a regression of Bug 843599.

Comment 14

[Sotaro Ikeda [:sotaro]]

When I disabled the HwComposer, preview was rendered normally.

> property_get("ro.display.colorfill", propValue, "0");
>  sUsingHwc = Preferences::GetBool("layers.composer2d.enabled",
>                                          atoi(propValue) == 1);

http://mxr.mozilla.org/mozilla-central/source/widget/gonk/nsWindow.cpp#196

Comment 15

[Sotaro Ikeda [:sotaro]]

This bug looks like dup of Bug 881460.

Comment 16

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #14)
> When I disabled the HwComposer, preview was rendered normally.
> 
> > property_get("ro.display.colorfill", propValue, "0");
> >  sUsingHwc = Preferences::GetBool("layers.composer2d.enabled",
> >                                          atoi(propValue) == 1);
> 
> http://mxr.mozilla.org/mozilla-central/source/widget/gonk/nsWindow.cpp#196

Sorry, was is my misunderstanding. On buri device, I can see camera preview even when hw compose is enabled.

Comment 17

[Mike Habicher [:mikeh] (high bugzilla latency)]

Okay, so when I apply the three patches in bug 881460, the camera viewfinder on Inari works properly! These are:
- patch from bug 881460 comment 7;
- patch from bug 881460 comment 16; and
- patch from bug 881460 comment 17.

Will confirm for Unagi as well.

Comment 18

[Milan Sreckovic [:milan] (needinfo for best results)]

I was trying to bisect this, but realized that my unagi doesn't actually let me run the camera in the first place since about a week ago.

Comment 19

[Mike Habicher [:mikeh] (high bugzilla latency)]

Comment 17: false alarm. Too much juggling today and I mixed up m-c and b2g18 builds. Even with the patches listed above, the camera viewfinder is still broken on m-c.

Comment 20

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to Milan Sreckovic [:milan] from comment #18)
>
> I was trying to bisect this, but realized that my unagi doesn't actually let
> me run the camera in the first place since about a week ago.

Milan, you need the patch from bug 879478 to keep the camera from crashing. That should be enough to let you see the garbled display.

Comment 21

[Randell Jesup [:jesup] (needinfo me)]

It appears that there's a timing issue here:

When viewing via gUM (http://mozilla.github.com/webrtc-landing/gum_test.html), I see green garbage, but if I press and hold the home button, the display starts sometimes showing green, sometimes showing the right video.  Then when it transitions to the captured image, it's correct.  This flickering of correct video implies to me it's a timing issue.

(note: blocking- means it doesn't block desktop landing)

Comment 22

[Sotaro Ikeda [:sotaro]]

It seems like current implementation heavily depends on ics_strawberry's(hamachi/buri/leo) genlock's characteristic.

Comment 23

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Ah ha.  The implementation requires a proper driver and genlock implementation.  I didn't realize that the devices that were talked about here were one of the broken ones, driver-wise.  If it works with devices with the right driver/genlock, then the bug is to get those devices upgraded to the right Qualcomm base.

Comment 24

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #23)
>
> If it works with devices with the right driver/genlock, then the bug is to get those
> devices upgraded to the right Qualcomm base.

Going with comment 22, it sounds like those are exactly the devices I don't have. Sotaro, can you confirm that the viewfinder is fine on hamachi/leo?

Comment 25

[Sotaro Ikeda [:sotaro]]

(In reply to Mike Habicher [:mikeh] from comment #24)
> (In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #23)
> Going with comment 22, it sounds like those are exactly the devices I don't
> have. Sotaro, can you confirm that the viewfinder is fine on hamachi/leo?

My buri was bricked yesterday. I can confirm on leo. I am going to confirm leo device.

Comment 26

[Sotaro Ikeda [:sotaro]]

I borrowed hamachi from :vlad. Thanks! I will check also hamachi.

Comment 27

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #23)
> Ah ha.  The implementation requires a proper driver and genlock
> implementation.  I didn't realize that the devices that were talked about
> here were one of the broken ones, driver-wise.  If it works with devices
> with the right driver/genlock, then the bug is to get those devices upgraded
> to the right Qualcomm base.

Just to be clear - is there a path of getting unagis working with the camera again that doesn't involve us mailing the units back to a magical place where drivers and genlock get installed?

Comment 28

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #26)
> I borrowed hamachi from :vlad. Thanks! I will check also hamachi.

I confirmed that the camera preview works on master hamachi.

Comment 29

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #28)
> (In reply to Sotaro Ikeda [:sotaro] from comment #26)
> > I borrowed hamachi from :vlad. Thanks! I will check also hamachi.
> 
> I confirmed that the camera preview works on master hamachi.

But preview update is not smooth.

Comment 30

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #28)
> (In reply to Sotaro Ikeda [:sotaro] from comment #26)
> > I borrowed hamachi from :vlad. Thanks! I will check also hamachi.
> 
> I confirmed that the camera preview works on master hamachi.

I applied a patch from Bug 882328 to run camera app correctly.

Comment 31

[Sotaro Ikeda [:sotaro]]

Leo device did not work correctly on master. On boot end, FTU was shown and I pushed "next" button, but FTU did not move to next page.

Comment 32

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to Milan Sreckovic [:milan] from comment #27)
> (In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #23)
> > Ah ha.  The implementation requires a proper driver and genlock
> > implementation.  I didn't realize that the devices that were talked about
> > here were one of the broken ones, driver-wise.  If it works with devices
> > with the right driver/genlock, then the bug is to get those devices upgraded
> > to the right Qualcomm base.
> 
> Just to be clear - is there a path of getting unagis working with the camera
> again that doesn't involve us mailing the units back to a magical place
> where drivers and genlock get installed?

Yes, just flashing a new system image with proper drivers.

Comment 33

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #31)
> Leo device did not work correctly on master. On boot end, FTU was shown and
> I pushed "next" button, but FTU did not move to next page.

By reverting Bug 852400, leo device's touch screen works correctly. Confirmed camera preview shows correctly. But preview update is not smooth as hamachi.

Comment 34

[Dale Harvey (:daleharvey)]

> Yes, just flashing a new system image with proper drivers.

Where can we get this?

Seen the issue today with a unagi on https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-central-unagi-eng/

Comment 35

[Dale Harvey (:daleharvey)]

> Just to be clear - is there a path of getting unagis working with the camera again that doesn't involve us mailing the units back to a magical place where drivers and genlock get installed?

You can install https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/latest/ and it works (with gaia master on top of b2g18 gecko things seems to be working too)

Comment 37

[Allen Maxwell]

This issue is occurring on v1.2.0 Unagi Mozilla RIL:
Build ID: 20130626031243
Gecko: http://hg.mozilla.org/mozilla-central/rev/cc80aa0c7c13
Gaia: 1949b5745b1fcda8600c544370e7162f136df4c8
Platform Version: 25.0a1

-Fails Core Sanity Check-
recommending escalation to Pri 2

Comment 38

[Dietrich Ayala (:dietrich)]

Not blocking because not happening on firefox os 1.1 on launch devices. please renominate if we're mistaken in that.

Comment 39

[Robert Kaiser]

This is also occurring on GP Peak with m-c/master.

Comment 40

[Jonathan Kew [:jfkthame]]

(In reply to Robert Kaiser (:[e-mail]) from comment #39)
> This is also occurring on GP Peak with m-c/master.

Confirmed here too on GP Peak, using local build of Gaia master + Gecko m-c. The camera viewfinder is totally green (I'm not seeing any significant noise, even), although the correct image usually flashes momentarily when first activating the camera, taking a photo, etc., and the photos themselves are correct (not green).

Comment 41

[Allen Maxwell]

per comment 37 and bug 887380; recommending escalation to Pri 2
For comment 38; the incorrect flag was thrown for this issue, since it is v1.2

Comment 43

[Juan Gomez [:_AtilA_] (CET/CEST)]

Attachment: logcat output for camera in preview mode.

I made a debug build and activated NSPR logging for the camera module.
The attached log has been caputred while the camera app was in preview mode.
There is a huge amount of information in the log, but I would like to highlight these lines:
 - E/mm-camera(  142): config_proc_MSG_ID_SOF_ACK: Error computing timestamps for EIS
 This is repeated along the entire log, it's being thrown by the camera HAL, and not really sure if it's related with our problem.

 - E/mm-libcamera2(  126): mm_camera_stream_qbuf: VIDIOC_QBUF error = -1, stream type=1
 - E/mm-libcamera2(  126): mm_camera_stream_util_buf_done: Error Trying to free second time?(idx=0) count=0, stream type=1
 - E/QCameraHWI_Preview(  126): BUF DONE FAILED
 These are being thrown by the HAL too, but just one time (line 1313), at the very beginning of the preview mode. As we don't have camera HAL source code, I cannot ensure it, but chances are that the first line is thrown by something like that:
rc = ioctl(stream->fd, VIDIOC_QBUF, &buffer);
IIUC this code tries to enqueue a buffer in the driver's incomming queue but it can't cause there's something wrong (unexpected format? bad user pointer?) with the gralloc buffer we are sharing with it.

Comment 44

[Allen Maxwell]

Attachment: Screenshot

There is no longer a green screen, but what appears to be a screen resolution issue when using camera app. 

Ref: CameraApp.png; CameraLog.txt

Build ID: 20130708031316
Gecko: http://hg.mozilla.org/mozilla-central/rev/17a47dcef75d
Gaia: b6268c5e22d5ab4dc7a74638e3c1b830e6f13d3d
Platform Version: 25.0a1

Comment 45

[Allen Maxwell]

Attachment: LogCat of issue

Comment 46

[Mike Habicher [:mikeh] (high bugzilla latency)]

Re Comment 44 - it looks like something has gone horribly wrong with the Camera app's response to the orientation sensor.

Comment 47

[Juan Gomez [:_AtilA_] (CET/CEST)]

Now the preview window is black on Peak.

Gecko: 33510110c092162e4d6218e98d378fc9c39ebf63 (9th - July)
Gaia: 7c5f93c12215a720e8b6da7406562ba045424eb6 (9th - July)

Comment 48

[Hubert Figuiere [:hub]]

Same here on Keon.

Comment 49

[Allen Maxwell]

Attachment: CameraGallerySwitching.png

Issue still occurring on:
Unagi v1.2.0 Mozilla RIL
Build ID: 20130718030209
Gecko: http://hg.mozilla.org/mozilla-central/rev/f26e4c26ce4a
Gaia: 4ec7c428f6a63a44f888ea6f6ade0385c89ae305
Platform Version: 25.0a1

1) Open Camera app

Result:
View finder is all black screen

Additionally:
2) Take picture
3) View thumbnail
4) Select camera

Result:
Camera view is distorted

Comment 50

[Allen Maxwell]

Attachment: LogCat of issue

Comment 51

[Juan Gomez [:_AtilA_] (CET/CEST)]

Allen, the attachment you provide in Comment 50 doesn't show anything about the issue being discussed here, isn't it?
I can only see GPS stuff in there...

Comment 52

[Allen Maxwell]

Attachment: LogCat of issue

In response to comment 51. I hope this has the needed data.

Comment 53

[Juan Gomez [:_AtilA_] (CET/CEST)]

Bug 858914, it's about to land soon and I think that it's going to change pices of code that are very related to this bug.. so let's see what happens when it lands..

Comment 54

[Chris Kreinbring [:CKreinbring]]

The camera viewfinder is still filled with green corruption on Unagi 1.2 mozilla RIL.

Build ID: 20130806070205
Gecko: http://hg.mozilla.org/mozilla-central/rev/a0dd80f800e2
Gaia: 49b1f68b0e2152557bb7c2defb2a5f9d7a0648c6
Platform Version: 26.0a1

Comment 55

[Milan Sreckovic [:milan] (needinfo for best results)]

I would expect it to be green - see Comment 23 in this bug.

Comment 56

[Sotaro Ikeda [:sotaro]]

If the green happens by the genlock problem, the green should not happen when gralloc buffer handling in gfx layer is correct. Current master does not handle it correctly. On master, gralloc buffers are released to camera hal too early timing.

Comment 57

[Mike Habicher [:mikeh] (high bugzilla latency)]

Is anyone looking at how to fix this? If not, can anyone point me to about where the problem might be, and I can take a look?

Comment 58

[Milan Sreckovic [:milan] (needinfo for best results)]

The lifetime is being handled in bug 858914. In the meantime, Buri/Hamachi works.

Comment 59

[Jason Smith [:jsmith]]

No longer a smoketest blocker - we've officially discontinued master + central support with unagi for daily smoketests.

Comment 60

[Sotaro Ikeda [:sotaro]]

From Bug 907745 comment #23, even if Bug 907745 and Bug 858914 are fixed, the viewfinder's problem seems not fixed.

Comment 61

[Sotaro Ikeda [:sotaro]]

From comment #60, this problem seems not depend on Bug 858914 and Bug 907745

Comment 62

[Sotaro Ikeda [:sotaro]]

attachment 803362 [details] [diff] [review] in Bug 912134 fixes the unagi's camera preview corruption problem. Fixes of Bug 913821 and Bug 901224 are also necessary.

Comment 63

[Juan Gomez [:_AtilA_] (CET/CEST)]

All patches referenced by Sotaro in comment #62 were landed and I can confirm that lastest master builds fixes the camera preview on Peak.

Comment 64

[Jason Smith [:jsmith]]

(In reply to Juan Gomez [:_AtilA_] from comment #63)
> All patches referenced by Sotaro in comment #62 were landed and I can
> confirm that lastest master builds fixes the camera preview on Peak.

Can someone confirm this fixed on the latest 1.2 build of unagi?

Comment 65

[Natalya Kot [:nkot]]

Have tested on the latest 1.2 Unagi build:
- viewfinder works properly on the first launch
- sometimes after taking a few pics the camera freezes
- it is required to kill the app or restart device to fix the issue

I'll look up if there is a bug to track this second issue, will file a new one if there is none 

Build ID: 20130916040205
Gecko: http://hg.mozilla.org/mozilla-central/rev/c4bcef90cef9
Gaia: a0079597d510ce8ea0b9cbb02c506030510b9eeb
Platform Version: 26.0a1

Comment 66

[Jason Smith [:jsmith]]

(In reply to nkot from comment #65)
> Have tested on the latest 1.2 Unagi build:
> - viewfinder works properly on the first launch
> - sometimes after taking a few pics the camera freezes
> - it is required to kill the app or restart device to fix the issue
> 
> I'll look up if there is a bug to track this second issue, will file a new
> one if there is none 
> 
> Build ID: 20130916040205
> Gecko: http://hg.mozilla.org/mozilla-central/rev/c4bcef90cef9
> Gaia: a0079597d510ce8ea0b9cbb02c506030510b9eeb
> Platform Version: 26.0a1

That implies the issue here is fixed - you would see a green viewfinder if this was still reproducing. Can you file a separate bug for the issue you found?

Comment 67

[Mike Habicher [:mikeh] (high bugzilla latency)]

Confirmed: the unagi viewfinder now works with m-c:dc909122bcf5. Thanks, sotaro!

Comment 68

[Natalya Kot [:nkot]]

have filed bug 916924 to track the issue mentioned in comment 65

Comment 69

[Robert Kaiser]

FWIW, viewfinder also works fine now on the Peak with current Nightly builds.

Comment 70

[Hubert Figuiere [:hub]]

yep, it works on todays nightly Keon from GP