[camera] camcorder crashes/hangs on L ports after recording

RESOLVED DUPLICATE of bug 1120780

Status

()

RESOLVED DUPLICATE of bug 1120780
4 years ago
4 years ago

People

(Reporter: ikumar, Unassigned)

Tracking

({crash})

unspecified
ARM
Gonk (Firefox OS)
crash
Points:
---

Firefox Tracking Flags

(blocking-b2g:2.2+)

Details

(Whiteboard: [b2g-crash][caf-crash 519][caf priority: p2][CR 798548])

Attachments

(15 attachments)

234.02 KB, text/plain
Details
124.26 KB, text/plain
Details
245.29 KB, text/plain
Details
100.90 KB, text/plain
Details
276.26 KB, text/plain
Details
101.48 KB, text/plain
Details
848 bytes, text/plain
Details
257.87 KB, text/plain
Details
106.94 KB, text/plain
Details
283.32 KB, text/plain
Details
154.95 KB, text/plain
Details
275.47 KB, text/plain
Details
105.41 KB, text/plain
Details
239.39 KB, text/plain
Details
291.93 KB, text/plain
Details
(Reporter)

Description

4 years ago
STR:
1. Start camera app
2. Switch to camcorder
3. Record a video
4. Stop the recording
Camera app hangs or crashes.
This is quite easily reproducible in 2-3 tries.
This issue does not reproduce on KK based devices.
(Reporter)

Updated

4 years ago
blocking-b2g: --- → 2.2?
(Reporter)

Comment 1

4 years ago
Created attachment 8564433 [details]
crash 1 stacktrace
(Reporter)

Comment 2

4 years ago
Created attachment 8564434 [details]
crash 1 logs
(Reporter)

Comment 3

4 years ago
Created attachment 8564436 [details]
crash 2 stacktrace
(Reporter)

Comment 4

4 years ago
Created attachment 8564437 [details]
crash 2 logs
(Reporter)

Comment 5

4 years ago
Sotaro - can you please take a quick look at the attachments to see if you have any clues?
Blocks: 1063044
Flags: needinfo?(sotaro.ikeda.g)
The following log seems to related to the crash. "1515870810" is "5A5A5A5A" in HEX. It say that the accessed memory is already freed. info->mStatus is OMXCodec's internal member. From it, the crash might be caused by OMXCodec's problem.

> 01-01 23:17:22.943  1243  1663 F OMXCodec: frameworks/av/media/libstagefright/OMXCodec.cpp:4420 CHECK_EQ( (int)info->mStatus,(int)OWNED_BY_US) failed: 1515870810 vs. 0
Flags: needinfo?(sotaro.ikeda.g)

Comment 7

4 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #6)
> The following log seems to related to the crash. "1515870810" is "5A5A5A5A"
> in HEX. It say that the accessed memory is already freed. info->mStatus is
> OMXCodec's internal member. From it, the crash might be caused by OMXCodec's
> problem.
> 
> > 01-01 23:17:22.943  1243  1663 F OMXCodec: frameworks/av/media/libstagefright/OMXCodec.cpp:4420 CHECK_EQ( (int)info->mStatus,(int)OWNED_BY_US) failed: 1515870810 vs. 0


Sotaro,

Are you saying this is an issue in CAF's code?

Thanks,
Mike
Flags: needinfo?(sotaro.ikeda.g)
(In reply to Mike Lee [:mlee] from comment #7)
> (In reply to Sotaro Ikeda [:sotaro] from comment #6)
> > The following log seems to related to the crash. "1515870810" is "5A5A5A5A"
> > in HEX. It say that the accessed memory is already freed. info->mStatus is
> > OMXCodec's internal member. From it, the crash might be caused by OMXCodec's
> > problem.
> > 
> > > 01-01 23:17:22.943  1243  1663 F OMXCodec: frameworks/av/media/libstagefright/OMXCodec.cpp:4420 CHECK_EQ( (int)info->mStatus,(int)OWNED_BY_US) failed: 1515870810 vs. 0
> 
> 
> Sotaro,
> 
> Are you saying this is an issue in CAF's code?

It is just one possibility. At this time, it seems more possible than other possibilities.
Flags: needinfo?(sotaro.ikeda.g)

Comment 9

4 years ago
NI back to Inder based on Sotaro's input on comment 8
Flags: needinfo?(ikumar)
(Reporter)

Comment 10

4 years ago
I have tried multiple times to reproduce this issue in Android with the same code base and we don't see this issue there.
The crash does not reproduce on KK based devices with v2.2 either so essentially points to L related integration issue in FirefoxOS.

To be fair, it's possible the issue is in OMXCodec and getting exposed with FirefoxOS but we would appreciate help to figure it out.

What's interesting is that the crash is manifesting with different call stack every time.
Flags: needinfo?(ikumar)
(Reporter)

Comment 11

4 years ago
Created attachment 8566853 [details]
crash 3 stacktrace
(Reporter)

Comment 12

4 years ago
Created attachment 8566854 [details]
crash 3 logs

Here is another one.
(In reply to Inder from comment #10)
> I have tried multiple times to reproduce this issue in Android with the same
> code base and we don't see this issue there.
> The crash does not reproduce on KK based devices with v2.2 either so
> essentially points to L related integration issue in FirefoxOS.

On android, when we do recording, hw codec instance and MPEG4Writer is in same process. But in b2g, the hw codec instance is in mediaserver, but the MPEG4Writer is in a b2g-content process. That might make a difference.

 
> To be fair, it's possible the issue is in OMXCodec and getting exposed with
> FirefoxOS but we would appreciate help to figure it out.
> 
> What's interesting is that the crash is manifesting with different call
> stack every time.

The crash call stack seems not work to fix this kind of problem. We might need to track all function call sequence of OMXCodec and related objects. mozilla side still do not have a environment to do debugging :-( I received the device, but still could not flash Firefox OS to it...
(In reply to Sotaro Ikeda [:sotaro] from comment #13)
> 
> I received the device, but still could not flash Firefox OS to it...

Taiwan office is closed until 2/23 because of chinese new year. After that they are going to work to address the ROM write problem with a partner.

Comment 15

4 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #13)
  
> > To be fair, it's possible the issue is in OMXCodec and getting exposed with
> > FirefoxOS but we would appreciate help to figure it out.
> > 
> > What's interesting is that the crash is manifesting with different call
> > stack every time.
> 
> The crash call stack seems not work to fix this kind of problem. We might
> need to track all function call sequence of OMXCodec and related objects.
> mozilla side still do not have a environment to do debugging :-( I received
> the device, but still could not flash Firefox OS to it...

Inder, 

Could you or someone from your team try to step through the functions and debug? I spoke to Sotaro offline and since the crash logs show nothing about the root-cause and we don't have the right working device, it becomes hard to debug this issue by weeding through the logs and stacktrace. He was saying that just enabling OMXCodec debug log may not provide enough info to analyze this problem. You may need to track all related functions' of OMXCodec enter and exit and see what is going on. At this point (since Moz doesn't have a working hardware yet), it looks like you guys are best equipped to dig in and figure out the problem. So, please try Sotaro's suggestion. 

Thanks
Hema
Flags: needinfo?(ikumar)
(Reporter)

Comment 16

4 years ago
We debugged with our video team on this and it turns out the issue is because we are using OMXCodec for camcorder recording. MediaRecorder on Android L uses MediaCodec for encoding audio and video instead of OMXCodec so we need to switch gecko to use MediaCodec instead.

We see that Bug 1033935 already tries to enable MediaCodec for v2.2 and it has all the approvals too. Not sure what it's waiting for.
Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we need all those 4 bug fixes uplifted to v2.2.
Flags: needinfo?(ikumar)

Updated

4 years ago
Whiteboard: [CR 798548]

Updated

4 years ago
Whiteboard: [CR 798548] → [caf priority: p2][CR 798548]

Comment 17

4 years ago
Adding dependency bug based on previous comments from Inder

NI Bhavana/Mike Lee to follow up on uplift status on bugs that are required by CAF: https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4
Component: Gaia::Camera → Video/Audio
Depends on: 1033935
Flags: needinfo?(mlee)
Flags: needinfo?(bbajaj)
Product: Firefox OS → Core
(In reply to Inder from comment #16)
> 
> We see that Bug 1033935 already tries to enable MediaCodec for v2.2 and it
> has all the approvals too. Not sure what it's waiting for.
> Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we
> need all those 4 bug fixes uplifted to v2.2.

Bug 1033935 is a bug for playback not for recording.
(In reply to Inder from comment #16)
> We debugged with our video team on this and it turns out the issue is
> because we are using OMXCodec for camcorder recording. MediaRecorder on
> Android L uses MediaCodec for encoding audio and video instead of OMXCodec
> so we need to switch gecko to use MediaCodec instead.
> 
> We see that Bug 1033935 already tries to enable MediaCodec for v2.2 and it
> has all the approvals too. Not sure what it's waiting for.
> Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we
> need all those 4 bug fixes uplifted to v2.2.

Camera recording via mozCamera api uses OMXCodec for a long time because how to recording is almost same to android. And the code also re-uses android's source code.
(In reply to Sotaro Ikeda [:sotaro] from comment #19)
> 
> Camera recording via mozCamera api uses OMXCodec for a long time because how
> to recording is almost same to android. And the code also re-uses android's
> source code.

These codes were just borrowed from android aosp code. Therefore, if we want to record via mozCamera api, it seems easier just to get android AOSP Lollipop as well.
From comment 18 and comment 20, the following fix is not related to recording problem.

> Also, based on https://bugzilla.mozilla.org/show_bug.cgi?id=1033935#c4 we
> need all those 4 bug fixes uplifted to v2.2.
About camera recording, it might better to get input from the people who works for b2g porting to L.
Vincent Liu, John Lin, can you comment to comment 23?
Flags: needinfo?(vliu)

Updated

4 years ago
Flags: needinfo?(jolin)

Comment 25

4 years ago
When I was porting Android L on Nexus 5, I also got similar issues that the camera got hang after camera recording. 

1. When stoping the recording, video track would block at pthread_join(). You can read the below message for detailed information.
   https://bugzilla.mozilla.org/show_bug.cgi?id=1114477#c13

2. "invalid color converter" message pops up when generating thumbnail. You may try the blow WIP to make sure it helps for this bug.
   https://bugzilla.mozilla.org/show_bug.cgi?id=1120780#c3
Then, about the crash, attachment 8543814 [details] [review] in Bug 1114477 seems to address the problem.

Updated

4 years ago
Whiteboard: [caf priority: p2][CR 798548] → [b2g-crash][caf-crash 519][caf priority: p2][CR 798548]

Updated

4 years ago
Keywords: crash
Created attachment 8568933 [details]
EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.083
Created attachment 8568934 [details]
decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.083
Flags: needinfo?(jolin)
Created attachment 8569155 [details]
EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.084
Created attachment 8569156 [details]
decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.084

Updated

4 years ago
Flags: needinfo?(mlee)

Updated

4 years ago
blocking-b2g: 2.2? → 2.2+
Flags: needinfo?(bbajaj)
(Reporter)

Comment 33

4 years ago
What solution are we looking into for v2.2? Do we want to stick with OMXCodec with the change mentioned in comment 25 or updating the implementation as mentioned in comment 22 to switch to MediaCodec?
I am also following up with video folks to see if there are any side effects with changing android framework to keep using OMXCodec.
Created attachment 8569680 [details]
EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.085
Created attachment 8569681 [details]
decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.085
(In reply to Inder from comment #33)
> What solution are we looking into for v2.2? Do we want to stick with
> OMXCodec with the change mentioned in comment 25 or updating the
> implementation as mentioned in comment 22 to switch to MediaCodec?
> I am also following up with video folks to see if there are any side effects
> with changing android framework to keep using OMXCodec.

If there is no problem/side effects of using OMXCodec, using OMXCodec seems better than MediaCodec.
(In reply to Sotaro Ikeda [:sotaro] from comment #37)
> 
> If there is no problem/side effects of using OMXCodec, using OMXCodec seems
> better than MediaCodec.

The code modification could become minimum and stable source code in b2g.
Created attachment 8570357 [details]
EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.086
Created attachment 8570358 [details]
decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.086
(Reporter)

Comment 42

4 years ago
I have landed a patch for MPEG4Writer similar to comment 26 in our local builds to enable OMXCodec during  recording and it's there since AU 89+. Let's see if this issue reproduces again.

Updated

4 years ago
Flags: needinfo?(vliu)

Comment 43

4 years ago
could you help on this?
Flags: needinfo?(slee)

Comment 44

4 years ago
As comment 42, asking QA to verify whether the bug still exists.
Flags: needinfo?(slee)
Keywords: qawanted
Created attachment 8576145 [details]
1133147-logcat-20150311.txt

Attempted to reproduce the steps in comment 0 of this bug on a Nexus 5 device flashed with the latest b2g v2.2 build. The Camera app hangs after creating a video, in the sense that all its buttons stop responding. If the Cards View is launched by long pressing the Home button and the Camera app is brought back as the main app, its buttons start responding again.

Steps to Reproduce
1. Start camera app
2. Switch to camcorder
3. Record a video
4. Stop the recording

Expected Results
Tapping on any other button works as expected.

Actual Results
Tapping on any other button does not produce any result, making it appear that the camera app is hung.

Steps to Reproduce (continued)
5. Switch to Cards View by long pressing the Software Home Button.
6. Switch back to the Camera app.

Expected Results
Tapping on any other button works as expected.

Actual Results
Tapping on any other button works as expected. At no point does the Camera app crash. Taking multiple photographs works as expected and produces no hangs or crashes. (See adb logcat attached as 1133147-logcat-20150311.txt)

Test Environment
Device-Name     hammerhead
Software        b2 2.2.0.0-prerelease 
Gaia-Rev        3f032238a52f08e4c2f68a47ad065a96eb22d470
Gecko-Rev       https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/004fa1cb1dd4
Build-ID        20150311002522
Version         37.0
FW-Release      5.0
FW-Incremental  eng.cltbld.20150311.040259
FW-Date         Wed Mar 11 04:03:14 EDT 2015
Bootloader      HHZ11n
(Reporter)

Updated

4 years ago
Blocks: 1142220
(Reporter)

Updated

4 years ago
No longer blocks: 1142220
Removing tag - See comment 45.
Flags: needinfo?(ktucker)
Keywords: qawanted
QA Whiteboard: [QAnalyst-Triage+]
Flags: needinfo?(ktucker)

Comment 47

4 years ago
(In reply to Parul Mathur [:pragmatic] from comment #45)
> Created attachment 8576145 [details]
> 1133147-logcat-20150311.txt
> 
> Attempted to reproduce the steps in comment 0 of this bug on a Nexus 5
> device flashed with the latest b2g v2.2 build. The Camera app hangs after
> creating a video, in the sense that all its buttons stop responding. If the
> Cards View is launched by long pressing the Home button and the Camera app
> is brought back as the main app, its buttons start responding again.
> 
> Steps to Reproduce
> 1. Start camera app
> 2. Switch to camcorder
> 3. Record a video
> 4. Stop the recording
> 
> Expected Results
> Tapping on any other button works as expected.
> 
> Actual Results
> Tapping on any other button does not produce any result, making it appear
> that the camera app is hung.
> 
> Steps to Reproduce (continued)
> 5. Switch to Cards View by long pressing the Software Home Button.
> 6. Switch back to the Camera app.
> 
> Expected Results
> Tapping on any other button works as expected.
> 
> Actual Results
> Tapping on any other button works as expected. At no point does the Camera
> app crash. Taking multiple photographs works as expected and produces no
> hangs or crashes. (See adb logcat attached as 1133147-logcat-20150311.txt)
> 
> Test Environment
> Device-Name     hammerhead
> Software        b2 2.2.0.0-prerelease 
> Gaia-Rev        3f032238a52f08e4c2f68a47ad065a96eb22d470
> Gecko-Rev      
> https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/004fa1cb1dd4
> Build-ID        20150311002522
> Version         37.0
> FW-Release      5.0
> FW-Incremental  eng.cltbld.20150311.040259
> FW-Date         Wed Mar 11 04:03:14 EDT 2015
> Bootloader      HHZ11n

On current Nexus 5 for v2.2 branch, this hang is expected since the patch of bug 1120780 doesn't land into it.

Comment 48

4 years ago
(In reply to Vincent Liu[:vliu] from comment #47)
> (In reply to Parul Mathur [:pragmatic] from comment #45)
> > Created attachment 8576145 [details]
> > 1133147-logcat-20150311.txt
> > 
> > Attempted to reproduce the steps in comment 0 of this bug on a Nexus 5
> > device flashed with the latest b2g v2.2 build. The Camera app hangs after
> > creating a video, in the sense that all its buttons stop responding. If the
> > Cards View is launched by long pressing the Home button and the Camera app
> > is brought back as the main app, its buttons start responding again.
> > 
> > Steps to Reproduce
> > 1. Start camera app
> > 2. Switch to camcorder
> > 3. Record a video
> > 4. Stop the recording
> > 
> > Expected Results
> > Tapping on any other button works as expected.
> > 
> > Actual Results
> > Tapping on any other button does not produce any result, making it appear
> > that the camera app is hung.
> > 
> > Steps to Reproduce (continued)
> > 5. Switch to Cards View by long pressing the Software Home Button.
> > 6. Switch back to the Camera app.
> > 
> > Expected Results
> > Tapping on any other button works as expected.
> > 
> > Actual Results
> > Tapping on any other button works as expected. At no point does the Camera
> > app crash. Taking multiple photographs works as expected and produces no
> > hangs or crashes. (See adb logcat attached as 1133147-logcat-20150311.txt)
> > 
> > Test Environment
> > Device-Name     hammerhead
> > Software        b2 2.2.0.0-prerelease 
> > Gaia-Rev        3f032238a52f08e4c2f68a47ad065a96eb22d470
> > Gecko-Rev      
> > https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/004fa1cb1dd4
> > Build-ID        20150311002522
> > Version         37.0
> > FW-Release      5.0
> > FW-Incremental  eng.cltbld.20150311.040259
> > FW-Date         Wed Mar 11 04:03:14 EDT 2015
> > Bootloader      HHZ11n
> 
> On current Nexus 5 for v2.2 branch, this hang is expected since the patch of
> bug 1120780 doesn't land into it.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1120780

Updated

4 years ago
No longer blocks: 1063044

Updated

4 years ago
Blocks: 1063044

Updated

4 years ago
No longer blocks: 1063044

Updated

4 years ago
Blocks: 1063044

Updated

4 years ago
No longer blocks: 1063044

Updated

4 years ago
Blocks: 1063044

Updated

4 years ago
No longer blocks: 1063044
You need to log in before you can comment on or make changes to this bug.