1133147 - [camera] camcorder crashes/hangs on L ports after recording

Status: RESOLVED DUPLICATE of bug 1120780

(Core :: Audio/Video, defect)

Description

[Inder]

STR:
1. Start camera app
2. Switch to camcorder
3. Record a video
4. Stop the recording
Camera app hangs or crashes.
This is quite easily reproducible in 2-3 tries.
This issue does not reproduce on KK based devices.

Comment 1

[Inder]

Attachment: crash 1 stacktrace

Comment 2

[Inder]

Attachment: crash 1 logs

Comment 3

[Inder]

Attachment: crash 2 stacktrace

Comment 4

[Inder]

Attachment: crash 2 logs

Comment 5

[Inder]

Sotaro - can you please take a quick look at the attachments to see if you have any clues?

Comment 6

[Sotaro Ikeda [:sotaro]]

The following log seems to related to the crash. "1515870810" is "5A5A5A5A" in HEX. It say that the accessed memory is already freed. info->mStatus is OMXCodec's internal member. From it, the crash might be caused by OMXCodec's problem.

> 01-01 23:17:22.943  1243  1663 F OMXCodec: frameworks/av/media/libstagefright/OMXCodec.cpp:4420 CHECK_EQ( (int)info->mStatus,(int)OWNED_BY_US) failed: 1515870810 vs. 0

Comment 7

[Mike Lee [:mlee]]

(In reply to Sotaro Ikeda [:sotaro] from comment #6)
> The following log seems to related to the crash. "1515870810" is "5A5A5A5A"
> in HEX. It say that the accessed memory is already freed. info->mStatus is
> OMXCodec's internal member. From it, the crash might be caused by OMXCodec's
> problem.
> 
> > 01-01 23:17:22.943  1243  1663 F OMXCodec: frameworks/av/media/libstagefright/OMXCodec.cpp:4420 CHECK_EQ( (int)info->mStatus,(int)OWNED_BY_US) failed: 1515870810 vs. 0


Sotaro,

Are you saying this is an issue in CAF's code?

Thanks,
Mike

Comment 8

[Sotaro Ikeda [:sotaro]]

(In reply to Mike Lee [:mlee] from comment #7)
> (In reply to Sotaro Ikeda [:sotaro] from comment #6)
> > The following log seems to related to the crash. "1515870810" is "5A5A5A5A"
> > in HEX. It say that the accessed memory is already freed. info->mStatus is
> > OMXCodec's internal member. From it, the crash might be caused by OMXCodec's
> > problem.
> > 
> > > 01-01 23:17:22.943  1243  1663 F OMXCodec: frameworks/av/media/libstagefright/OMXCodec.cpp:4420 CHECK_EQ( (int)info->mStatus,(int)OWNED_BY_US) failed: 1515870810 vs. 0
> 
> 
> Sotaro,
> 
> Are you saying this is an issue in CAF's code?

It is just one possibility. At this time, it seems more possible than other possibilities.

Comment 9

[Hema Koka [:hema]]

NI back to Inder based on Sotaro's input on comment 8

Comment 10

[Inder]

I have tried multiple times to reproduce this issue in Android with the same code base and we don't see this issue there.
The crash does not reproduce on KK based devices with v2.2 either so essentially points to L related integration issue in FirefoxOS.

To be fair, it's possible the issue is in OMXCodec and getting exposed with FirefoxOS but we would appreciate help to figure it out.

What's interesting is that the crash is manifesting with different call stack every time.

Comment 11

[Inder]

Attachment: crash 3 stacktrace

Comment 12

[Inder]

Attachment: crash 3 logs

Here is another one.

Comment 13

[Sotaro Ikeda [:sotaro]]

(In reply to Inder from comment #10)
> I have tried multiple times to reproduce this issue in Android with the same
> code base and we don't see this issue there.
> The crash does not reproduce on KK based devices with v2.2 either so
> essentially points to L related integration issue in FirefoxOS.

On android, when we do recording, hw codec instance and MPEG4Writer is in same process. But in b2g, the hw codec instance is in mediaserver, but the MPEG4Writer is in a b2g-content process. That might make a difference.

 
> To be fair, it's possible the issue is in OMXCodec and getting exposed with
> FirefoxOS but we would appreciate help to figure it out.
> 
> What's interesting is that the crash is manifesting with different call
> stack every time.

The crash call stack seems not work to fix this kind of problem. We might need to track all function call sequence of OMXCodec and related objects. mozilla side still do not have a environment to do debugging :-( I received the device, but still could not flash Firefox OS to it...

Comment 14

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #13)
> 
> I received the device, but still could not flash Firefox OS to it...

Taiwan office is closed until 2/23 because of chinese new year. After that they are going to work to address the ROM write problem with a partner.

Comment 15

[Hema Koka [:hema]]

(In reply to Sotaro Ikeda [:sotaro] from comment #13)
  
> > To be fair, it's possible the issue is in OMXCodec and getting exposed with
> > FirefoxOS but we would appreciate help to figure it out.
> > 
> > What's interesting is that the crash is manifesting with different call
> > stack every time.
> 
> The crash call stack seems not work to fix this kind of problem. We might
> need to track all function call sequence of OMXCodec and related objects.
> mozilla side still do not have a environment to do debugging :-( I received
> the device, but still could not flash Firefox OS to it...

Inder, 

Could you or someone from your team try to step through the functions and debug? I spoke to Sotaro offline and since the crash logs show nothing about the root-cause and we don't have the right working device, it becomes hard to debug this issue by weeding through the logs and stacktrace. He was saying that just enabling OMXCodec debug log may not provide enough info to analyze this problem. You may need to track all related functions' of OMXCodec enter and exit and see what is going on. At this point (since Moz doesn't have a working hardware yet), it looks like you guys are best equipped to dig in and figure out the problem. So, please try Sotaro's suggestion. 

Thanks
Hema

Comment 16

[Inder]

We debugged with our video team on this and it turns out the issue is because we are using OMXCodec for camcorder recording. MediaRecorder on Android L uses MediaCodec for encoding audio and video instead of OMXCodec so we need to switch gecko to use MediaCodec instead.

We see that Bug 1033935 already tries to enable MediaCodec for v2.2 and it has all the approvals too. Not sure what it's waiting for.
Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we need all those 4 bug fixes uplifted to v2.2.

Comment 17

[Hema Koka [:hema]]

Adding dependency bug based on previous comments from Inder

NI Bhavana/Mike Lee to follow up on uplift status on bugs that are required by CAF: https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4

Comment 18

[Sotaro Ikeda [:sotaro]]

(In reply to Inder from comment #16)
> 
> We see that Bug 1033935 already tries to enable MediaCodec for v2.2 and it
> has all the approvals too. Not sure what it's waiting for.
> Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we
> need all those 4 bug fixes uplifted to v2.2.

Bug 1033935 is a bug for playback not for recording.

Comment 19

[Sotaro Ikeda [:sotaro]]

(In reply to Inder from comment #16)
> We debugged with our video team on this and it turns out the issue is
> because we are using OMXCodec for camcorder recording. MediaRecorder on
> Android L uses MediaCodec for encoding audio and video instead of OMXCodec
> so we need to switch gecko to use MediaCodec instead.
> 
> We see that Bug 1033935 already tries to enable MediaCodec for v2.2 and it
> has all the approvals too. Not sure what it's waiting for.
> Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we
> need all those 4 bug fixes uplifted to v2.2.

Camera recording via mozCamera api uses OMXCodec for a long time because how to recording is almost same to android. And the code also re-uses android's source code.

Comment 20

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #19)
> 
> Camera recording via mozCamera api uses OMXCodec for a long time because how
> to recording is almost same to android. And the code also re-uses android's
> source code.

These codes were just borrowed from android aosp code. Therefore, if we want to record via mozCamera api, it seems easier just to get android AOSP Lollipop as well.

Comment 21

[Sotaro Ikeda [:sotaro]]

From comment 18 and comment 20, the following fix is not related to recording problem.

> Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we
> need all those 4 bug fixes uplifted to v2.2.

Comment 22

[Sotaro Ikeda [:sotaro]]

In current gecko, GonkCameraSource and GonkRecorder are the classes that were borrwed from AOSP.

https://dxr.mozilla.org/mozilla-central/source/dom/camera/GonkRecorder.cpp
https://dxr.mozilla.org/mozilla-central/source/dom/camera/GonkCameraSource.cpp


From L AOSP, we might need to borrow StagefrightRecorder, CameraSource and SurfaceMediaSource classes.

http://androidxref.com/5.0.0_r2/xref/frameworks/av/media/libmediaplayerservice/StagefrightRecorder.cpp
http://androidxref.com/5.0.0_r2/xref/frameworks/av/media/libstagefright/CameraSource.cpp
http://androidxref.com/5.0.0_r2/xref/frameworks/av/media/libstagefright/MediaCodecSource.cpp

Comment 23

[Sotaro Ikeda [:sotaro]]

About camera recording, it might better to get input from the people who works for b2g porting to L.

Comment 24

[Sotaro Ikeda [:sotaro]]

Vincent Liu, John Lin, can you comment to comment 23?

Comment 25

[Vincent Liu[:vliu]]

When I was porting Android L on Nexus 5, I also got similar issues that the camera got hang after camera recording. 

1. When stoping the recording, video track would block at pthread_join(). You can read the below message for detailed information.
   https://bugzilla.mozilla.org/show_bug.cgi?id=1114477#c13

2. "invalid color converter" message pops up when generating thumbnail. You may try the blow WIP to make sure it helps for this bug.
   https://bugzilla.mozilla.org/show_bug.cgi?id=1120780#c3

Comment 26

[Sotaro Ikeda [:sotaro]]

Then, about the crash, attachment 8543814 [details] [review] in Bug 1114477 seems to address the problem.

Comment 27

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.083
Moz BuildID: 20150222002501
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.083.xml?h=release
Gecko Version: 37.0a2
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=be86a57c64529b651e67ba4ae87d70acffbc986a
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=7aa23764fe2edd6d8e5116d237b93a05dc591751
Patches: bug 1067629, bug 1125940, bug 1125418, bug 1091307, bug 1132528, bug 1134821

Comment 28

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.083

Comment 29

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.083

Comment 30

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.084
Moz BuildID: 20150223002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.084.xml?h=release
Gecko Version: 37.0a2
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=389542b71c89253c0d176d3b0bfb54e275c19bf1
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=84e31b298b5e1eb249a07083f2c26dcc2a6798d2
Patches: bug 1067629, bug 1125940, bug 1125418, bug 1091307, bug 1132528, bug 1134821

Comment 31

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.084

Comment 32

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.084

Comment 33

[Inder]

What solution are we looking into for v2.2? Do we want to stick with OMXCodec with the change mentioned in comment 25 or updating the implementation as mentioned in comment 22 to switch to MediaCodec?
I am also following up with video folks to see if there are any side effects with changing android framework to keep using OMXCodec.

Comment 34

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.085
Moz BuildID: 20150225002505
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.085.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=ca64f2fe145909f31af266b1730874051ba76c78
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=fe489912aac2f83b14e7286b8f92443f6c6b64a7
Patches: bug 1067629, bug 1132528, bug 1125418, bug 1091307, bug 1134821

Comment 35

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.085

Comment 36

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.085

Comment 37

[Sotaro Ikeda [:sotaro]]

(In reply to Inder from comment #33)
> What solution are we looking into for v2.2? Do we want to stick with
> OMXCodec with the change mentioned in comment 25 or updating the
> implementation as mentioned in comment 22 to switch to MediaCodec?
> I am also following up with video folks to see if there are any side effects
> with changing android framework to keep using OMXCodec.

If there is no problem/side effects of using OMXCodec, using OMXCodec seems better than MediaCodec.

Comment 38

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #37)
> 
> If there is no problem/side effects of using OMXCodec, using OMXCodec seems
> better than MediaCodec.

The code modification could become minimum and stable source code in b2g.

Comment 39

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.086
Moz BuildID: 20150225002505
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.086.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=ca64f2fe145909f31af266b1730874051ba76c78
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=fe489912aac2f83b14e7286b8f92443f6c6b64a7
Patches: bug 1067629, bug 1132528, bug 1125418, bug 1091307

Comment 40

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.086

Comment 41

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.086

Comment 42

[Inder]

I have landed a patch for MPEG4Writer similar to comment 26 in our local builds to enable OMXCodec during  recording and it's there since AU 89+. Let's see if this issue reproduces again.

Comment 43

[Bobby Chien]

could you help on this?

Comment 44

[StevenLee[:slee]]

As comment 42, asking QA to verify whether the bug still exists.

Comment 45

[Parul Mathur [:pragmatic] - out of office - please ni? [:bajaj] instead]

Attachment: 1133147-logcat-20150311.txt

Attempted to reproduce the steps in comment 0 of this bug on a Nexus 5 device flashed with the latest b2g v2.2 build. The Camera app hangs after creating a video, in the sense that all its buttons stop responding. If the Cards View is launched by long pressing the Home button and the Camera app is brought back as the main app, its buttons start responding again.

Steps to Reproduce
1. Start camera app
2. Switch to camcorder
3. Record a video
4. Stop the recording

Expected Results
Tapping on any other button works as expected.

Actual Results
Tapping on any other button does not produce any result, making it appear that the camera app is hung.

Steps to Reproduce (continued)
5. Switch to Cards View by long pressing the Software Home Button.
6. Switch back to the Camera app.

Expected Results
Tapping on any other button works as expected.

Actual Results
Tapping on any other button works as expected. At no point does the Camera app crash. Taking multiple photographs works as expected and produces no hangs or crashes. (See adb logcat attached as 1133147-logcat-20150311.txt)

Test Environment
Device-Name     hammerhead
Software        b2 2.2.0.0-prerelease 
Gaia-Rev        3f032238a52f08e4c2f68a47ad065a96eb22d470
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/004fa1cb1dd4
Build-ID        20150311002522
Version         37.0
FW-Release      5.0
FW-Incremental  eng.cltbld.20150311.040259
FW-Date         Wed Mar 11 04:03:14 EDT 2015
Bootloader      HHZ11n

Comment 46

[Pi Wei Cheng [:piwei] (inactive)]

Removing tag - See comment 45.

Comment 47

[Vincent Liu[:vliu]]

(In reply to Parul Mathur [:pragmatic] from comment #45)
> Created attachment 8576145 [details]
> 1133147-logcat-20150311.txt
> 
> Attempted to reproduce the steps in comment 0 of this bug on a Nexus 5
> device flashed with the latest b2g v2.2 build. The Camera app hangs after
> creating a video, in the sense that all its buttons stop responding. If the
> Cards View is launched by long pressing the Home button and the Camera app
> is brought back as the main app, its buttons start responding again.
> 
> Steps to Reproduce
> 1. Start camera app
> 2. Switch to camcorder
> 3. Record a video
> 4. Stop the recording
> 
> Expected Results
> Tapping on any other button works as expected.
> 
> Actual Results
> Tapping on any other button does not produce any result, making it appear
> that the camera app is hung.
> 
> Steps to Reproduce (continued)
> 5. Switch to Cards View by long pressing the Software Home Button.
> 6. Switch back to the Camera app.
> 
> Expected Results
> Tapping on any other button works as expected.
> 
> Actual Results
> Tapping on any other button works as expected. At no point does the Camera
> app crash. Taking multiple photographs works as expected and produces no
> hangs or crashes. (See adb logcat attached as 1133147-logcat-20150311.txt)
> 
> Test Environment
> Device-Name     hammerhead
> Software        b2 2.2.0.0-prerelease 
> Gaia-Rev        3f032238a52f08e4c2f68a47ad065a96eb22d470
> Gecko-Rev      
> https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/004fa1cb1dd4
> Build-ID        20150311002522
> Version         37.0
> FW-Release      5.0
> FW-Incremental  eng.cltbld.20150311.040259
> FW-Date         Wed Mar 11 04:03:14 EDT 2015
> Bootloader      HHZ11n

On current Nexus 5 for v2.2 branch, this hang is expected since the patch of bug 1120780 doesn't land into it.

Comment 48

[Bobby Chien]

(In reply to Vincent Liu[:vliu] from comment #47)
> (In reply to Parul Mathur [:pragmatic] from comment #45)
> > Created attachment 8576145 [details]
> > 1133147-logcat-20150311.txt
> > 
> > Attempted to reproduce the steps in comment 0 of this bug on a Nexus 5
> > device flashed with the latest b2g v2.2 build. The Camera app hangs after
> > creating a video, in the sense that all its buttons stop responding. If the
> > Cards View is launched by long pressing the Home button and the Camera app
> > is brought back as the main app, its buttons start responding again.
> > 
> > Steps to Reproduce
> > 1. Start camera app
> > 2. Switch to camcorder
> > 3. Record a video
> > 4. Stop the recording
> > 
> > Expected Results
> > Tapping on any other button works as expected.
> > 
> > Actual Results
> > Tapping on any other button does not produce any result, making it appear
> > that the camera app is hung.
> > 
> > Steps to Reproduce (continued)
> > 5. Switch to Cards View by long pressing the Software Home Button.
> > 6. Switch back to the Camera app.
> > 
> > Expected Results
> > Tapping on any other button works as expected.
> > 
> > Actual Results
> > Tapping on any other button works as expected. At no point does the Camera
> > app crash. Taking multiple photographs works as expected and produces no
> > hangs or crashes. (See adb logcat attached as 1133147-logcat-20150311.txt)
> > 
> > Test Environment
> > Device-Name     hammerhead
> > Software        b2 2.2.0.0-prerelease 
> > Gaia-Rev        3f032238a52f08e4c2f68a47ad065a96eb22d470
> > Gecko-Rev      
> > https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/004fa1cb1dd4
> > Build-ID        20150311002522
> > Version         37.0
> > FW-Release      5.0
> > FW-Incremental  eng.cltbld.20150311.040259
> > FW-Date         Wed Mar 11 04:03:14 EDT 2015
> > Bootloader      HHZ11n
> 
> On current Nexus 5 for v2.2 branch, this hang is expected since the patch of
> bug 1120780 doesn't land into it.