825986 - crash in android::GonkCameraSource::startCameraRecording

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect, P1)

Description

[Michael Vines [:m1] [:evilmachines]]

Attachment: decoded minidump of crash

New crash that has appeared in recent builds seen multiple times during overnight camera stability test.

Comment 1

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

void GonkCameraSource::startCameraRecording() {
    LOGV("startCameraRecording");
    CHECK_EQ(OK, GonkCameraHardware::StartRecording(mCameraHandle));

It looks like we're failing the assertion(?) here.  Should this really be an assert?  Is there no way for us to propagate error state?

Comment 2

[Mike Habicher [:mikeh] (high bugzilla latency)]

*yoink*

Comment 3

[Michael Vines [:m1] [:evilmachines]]

(ouch ouch ouch.  this crash just came up again tonight multiple times.)

Comment 4

[Mike Habicher [:mikeh] (high bugzilla latency)]

Any STR?

Comment 5

[Michael Vines [:m1] [:evilmachines]]

Looks like just open/close camcorder repeatedly.  I would attach the logcat but it's 1.7GB.  Can share it at the ww if needed.

Comment 6

[Mike Habicher [:mikeh] (high bugzilla latency)]

Just to clarify: do you mean open/close the camera app? or switch to/from camcorder mode? or start/stop recording?

Please bring the logcat to the WW and I can dig into it there.

Comment 7

[Michael Vines [:m1] [:evilmachines]]

I'll get those details for you asap.  But I think you can get started immediately as it looks like we are failing an assertion per comment 1.

Comment 8

[Michael Vines [:m1] [:evilmachines]]

(In reply to Mike Habicher [:mikeh] from comment #6)
> do you mean open/close the camera app? or switch to/from camcorder mode? or start/stop recording?

1. Launch camera app
2. Switch to camcorder mode
3. Record a 10s clip
4. Check that it was stored on the SD card
5. Navigate out of the camera app (which is not closed explicitly)
5. Goto 1

Comment 9

[Mike Habicher [:mikeh] (high bugzilla latency)]

Created a patch that propagates the StartRecording() return code, instead of just aborting on !OK.  To test, I made startCameraRecording() return DEAD_OBJECT instead of calling anything; that works, but when I switch back to photo mode after getting the error message in the UI, the camera crashes.

STR:
1. open camera
2. switch to video mode
3. press record button*
4. dismiss error message
5. switch to still mode
--- crash!

* with startCameraRecording() always returning DEAD_OBJECT.

Comment 10

[Mike Habicher [:mikeh] (high bugzilla latency)]

logcat:

E/QualcommCameraHardware(  436): stopPreview : after cancelling thumbnail buffer
E/QualcommCameraHardware(  436): stoppreview : display unlock
I/QualcommCameraHardware(  436): stopPreviewInternal E: 1
I/PRLog   (  436): 9763464[42507710]: queueBuffer: E
I/PRLog   (  436): 9763464[42507710]: GonkNativeWindow::getCurrentBuffer
I/PRLog   (  436): 9763464[42507710]: android::CameraGraphicBuffer::CameraGraphicBuffer(android::GonkNativeWindow*, uint32_t, uint32_t, mozilla::layers::SurfaceDescriptor):254 : this=448bf400
I/PRLog   (  436): 9763464[42507710]: bool mozilla::DOMCameraPreview::ReceiveFrame(void*, mozilla::ImageFormat, void (*)(mozilla::layers::Image*, void*, uint32_t, uint32_t)):176 : this=4425fdc0
I/PRLog   (  436): 9763464[42507710]: queueBuffer: X
I/PRLog   (  436): 9763464[42507710]: dequeueBuffer: E
I/PRLog   (  436): 9763464[42507710]: dequeueBuffer: returning slot=8 buf=442e1bf0 
I/PRLog   (  436): 9763464[42507710]: dequeueBuffer: X
I/PRLog   (  436): 9763464[42507710]: GonkNativeWindow::lockBuffer
E/QualcommCameraHardware(  436): Could not find the Frame
V/QualcommCameraHardware(  436):  error Cancelling preview buffers  
I/PRLog   (  436): 9763464[42507710]: cancelBuffer: slot=8
I/PRLog   (  436): 9763464[42507710]: queueBuffer: E
I/PRLog   (  436): 9763464[42507710]: GonkNativeWindow::getCurrentBuffer
I/PRLog   (  436): 9763464[42507710]: android::CameraGraphicBuffer::CameraGraphicBuffer(android::GonkNativeWindow*, uint32_t, uint32_t, mozilla::layers::SurfaceDescriptor):254 : this=448cd180
I/PRLog   (  436): 9763464[42507710]: bool mozilla::DOMCameraPreview::ReceiveFrame(void*, mozilla::ImageFormat, void (*)(mozilla::layers::Image*, void*, uint32_t, uint32_t)):176 : this=4425fdc0
I/PRLog   (  436): 9763464[42507710]: queueBuffer: X
I/PRLog   (  436): 9763464[42507710]: dequeueBuffer: E
I/PRLog   (  436): 9763464[42507710]: dequeueBuffer: returning slot=8 buf=442e1bf0 
I/PRLog   (  436): 9763464[42507710]: dequeueBuffer: X
I/PRLog   (  436): 9763464[42507710]: GonkNativeWindow::lockBuffer
E/QualcommCameraHardware(  436): Could not find the Frame
V/QualcommCameraHardware(  436):  error Cancelling preview buffers  
I/PRLog   (  436): 9763464[42507710]: cancelBuffer: slot=8
I/PRLog   (  436): 9763400[44626200]: GonkNativeWindow::returnBuffer: slot=6 (generation=1)
I/PRLog   (  436): 9763400[44626200]: virtual android::CameraGraphicBuffer::~CameraGraphicBuffer():259 : this=431cc780
I/PRLog   (  436): 9763400[44626200]: GonkNativeWindow::returnBuffer: slot=0 (generation=1)
I/PRLog   (  436): 9763400[44626200]: virtual android::CameraGraphicBuffer::~CameraGraphicBuffer():259 : this=431cc800
E/QualcommCameraHardware(  436): stopPreviewInternal, J_mCameraRunning = 0
E/QualcommCameraHardware(  436): stopPreviewInternal, before calling deinitpre mPreviewInitialized = 1
E/QualcommCameraHardware(  436): before calling deinitpreview
I/QualcommCameraHardware(  436): deinitPreview E
I/QualcommCameraHardware(  436): deinitPreview X
I/QualcommCameraHardware(  436): stopPreviewInternal: waiting for old frame thread to complete.
I/QualcommCameraHardware(  436): preview_thread X
V/QualcommCameraHardware(  436): relinquishBuffers: E 
I/PRLog   (  436): 9763528[42507710]: cancelBuffer: slot=0
I/Gecko   (  436): cancelBuffer: slot 0 is not owned by the client (state=0)
F/libc    (  436): Fatal signal 11 (SIGSEGV) at 0xa5a5a5a5 (code=1)

Comment 11

[Mike Habicher [:mikeh] (high bugzilla latency)]

bt from STR in comment 9:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 619.676]
0x42e1e778 in android::QualcommCameraHardware::relinquishBuffers() () from /home/mikeh/dev/mozilla/btg019/out/target/product/unagi/system/lib/hw/camera.msm7627a.so
(gdb) bt
#0  0x42e1e778 in android::QualcommCameraHardware::relinquishBuffers() () from /home/mikeh/dev/mozilla/btg019/out/target/product/unagi/system/lib/hw/camera.msm7627a.so
#1  0x40099144 in _normal_unlock (mutex=0x431cb440) at bionic/libc/bionic/pthread.c:973
#2  pthread_mutex_unlock (mutex=0x431cb440) at bionic/libc/bionic/pthread.c:1120
#3  0x431cb000 in ?? ()
#4  0x431cb000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Comment 12

[Mike Habicher [:mikeh] (high bugzilla latency)]

It looks like something in relinquishBuffers() isn't properly initialized when the call to startRecording() either fails or is never made.  I'm trying to track down exactly where the SIGSEGV is happening, but I only have a binary blob.  Would be handle if I could build QCH.cpp.

Comment 13

[Mike Habicher [:mikeh] (high bugzilla latency)]

Tracked down!

I can fix the assert on our end, but this has exposed an underlying bug in the driver constructor.  Specifically, kRecordBufferCount is being used in a loop before it is initialized; but since it is a static, the runtime sets it to zero, and the loop does nothing.  The result is an array of uninitialized pointers, which cause relinquishBuffers() to meltdown when a recorder object is destroyed.

This bug doesn't appear if startRecording() is called and returns _successfully_, since the pointers are given proper values in that function.

I will submit a patch to remove the assert.

Comment 14

[Mike Habicher [:mikeh] (high bugzilla latency)]

Attachment: Propagate error returned by StartRecording(), instead of asserting

Note the important bug which this exposes, as described in comment 13.

Comment 15

[Mike Habicher [:mikeh] (high bugzilla latency)]

Just to clarify comment 13: STR:
1. open the camera app
2. switch to video mode
3. press the start recording button

Comment 16

[Mike Habicher [:mikeh] (high bugzilla latency)]

Just to clarify comment 13, STR:
1. open the camera app
2. switch to video mode
3. press the start recording button
    * in order to trigger the underlying bug, this step must fail; when it does:
4. dismiss the error message
5. switch to picture mode
    * crash

So long as step 3 (and step 4) either succeeds or is skipped, the camera will not crash due to the underlying bug.

Comment 17

[Mike Habicher [:mikeh] (high bugzilla latency)]

try-server push: https://tbpl.mozilla.org/?tree=Try&rev=9dbd33d1896e

Comment 18

[Kan-Ru Chen [:kanru] (UTC+9)]

Comment on attachment 698715 [details] [diff] [review]
Propagate error returned by StartRecording(), instead of asserting

Review of attachment 698715 [details] [diff] [review]:
-----------------------------------------------------------------

looks good.

Comment 19

[Andrea Marchesini [:baku]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fbe1084946e7

Comment 20

[Mike Habicher [:mikeh] (high bugzilla latency)]

Confirmed: with the above patch and a fix for the driver constructor, failing to start video recording no longer sets up a latent crash.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/fbe1084946e7

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/d3400320ddb0