1307864 - Crash in mozilla::java::sdk::MediaCodec::DequeueOutputBuffer

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Audio/Video, defect, P1)

Description

[Marcia Knous [:marcia]]

[Tracking Requested - why for this release]: Nominating for tracking to keep on eye on this new signature.

This bug was filed from the Socorro interface and is 
report bp-0ab3f74e-869e-492f-946d-af85f2161005.
=============================================================

Seen while looking at crash-stats: http://bit.ly/2dSPwPG. New crash in beta - #14 top crash.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

This looks like a media bug. Here's one crash which has some logcat output:

https://crash-stats.mozilla.com/report/index/f2f708fc-492a-4a4c-9f1c-290ba2161001#tab-metadata

The last few lines before crashing:

10-01 16:20:46.077 28927 31089 I OMXClient: Using client-side OMX mux.
10-01 16:20:46.077 28927 31089 I         : new RKOMXPlugin
10-01 16:20:46.077 28927 31089 D ACodec  : MEDIA_MIMETYPE_AUDIO_AAC sampleRate 22050 wifidisplay_flag 0 info 9530de78
10-01 16:20:46.077 28927 31089 E OMXNodeInstance: OMX_GetExtensionIndex failed
10-01 16:20:46.097 28927 31095 I OMXClient: Using client-side OMX mux.
10-01 16:20:46.097 28927 31095 E ACodec  : Unable to instantiate a decoder for type 'video/avc'.
10-01 16:20:46.097 28927 31094 E MediaCodec: Codec reported an error. (omx error 0x80001003, internalError -2147483648)
10-01 16:20:46.147 28927 30984 W google-breakpad: ExceptionHandler::GenerateDump cloned child 
10-01 16:20:46.147 28927 30984 W google-breakpad: 31097
10-01 16:20:46.147 28927 30984 W google-breakpad: 
10-01 16:20:46.147 28927 30984 W google-breakpad: ExceptionHandler::SendContinueSignalToChild sent continue signal to child
10-01 16:20:46.147 31097 31097 W google-breakpad: ExceptionHandler::WaitForContinueSignal waiting for continue signal...

Comment 2

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hello Chris, Jean-Yves, this is a new crash in Fennec50. Could you please help find an owner? Thanks!

Comment 3

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

It's ranked #15 in Fennec 50 top crashers.

Comment 4

[Chris Pearce [:cpearce (Not reading bugmail)]]

Blake is managing Android video playback. Passing ni? to him.

Comment 5

[Blake Wu [:bwu][:blakewu]]

Chris, 
Thanks for letting me know. 

James, 
Please help this bug. 
Thanks!

Comment 6

[James Cheng[:JamesCheng]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #1)
> This looks like a media bug. Here's one crash which has some logcat output:
> 
> https://crash-stats.mozilla.com/report/index/f2f708fc-492a-4a4c-9f1c-
> 290ba2161001#tab-metadata
> 
> The last few lines before crashing:
> 
> 10-01 16:20:46.077 28927 31089 I OMXClient: Using client-side OMX mux.
> 10-01 16:20:46.077 28927 31089 I         : new RKOMXPlugin
> 10-01 16:20:46.077 28927 31089 D ACodec  : MEDIA_MIMETYPE_AUDIO_AAC
> sampleRate 22050 wifidisplay_flag 0 info 9530de78
> 10-01 16:20:46.077 28927 31089 E OMXNodeInstance: OMX_GetExtensionIndex
> failed
> 10-01 16:20:46.097 28927 31095 I OMXClient: Using client-side OMX mux.
> 10-01 16:20:46.097 28927 31095 E ACodec  : Unable to instantiate a decoder
> for type 'video/avc'.
> 10-01 16:20:46.097 28927 31094 E MediaCodec: Codec reported an error. (omx
> error 0x80001003, internalError -2147483648)
> 10-01 16:20:46.147 28927 30984 W google-breakpad:
> ExceptionHandler::GenerateDump cloned child 
> 10-01 16:20:46.147 28927 30984 W google-breakpad: 31097
> 10-01 16:20:46.147 28927 30984 W google-breakpad: 
> 10-01 16:20:46.147 28927 30984 W google-breakpad:
> ExceptionHandler::SendContinueSignalToChild sent continue signal to child
> 10-01 16:20:46.147 31097 31097 W google-breakpad:
> ExceptionHandler::WaitForContinueSignal waiting for continue signal...

Hi kats,

I would like to know where did you get the logcat from those links.

I cannot find out the logcat information...

Thanks.

Comment 7

[James Cheng[:JamesCheng]]

From the logcat snippet, I can see we cannot create the decoder for video/avc which is not a normal situation since H264 is very common. The error 0x80001003 means "No component with the specified name string was found" [1]

But that error is not related to crash I think.

From the call stack[2], it seems the "audio" decoder can be created but bump into crash when invoking DequeueOutputBuffer.

This is where C++ wants to invoke JAVA code throgh JNI but I have no idea why  

```__aeabi_fcmpgt``` got involved with the callstack and crash in libdvm.so.


Hi Jim,

Do you have any idea about this point(__aeabi_fcmpgt and JNI)?

Thanks.

[1]
http://androidxref.com/6.0.1_r10/xref/frameworks/native/include/media/openmax/OMX_Core.h#157

[2]
https://crash-stats.mozilla.com/report/index/f2f708fc-492a-4a4c-9f1c-290ba2161001#tab-details

Comment 8

[James Cheng[:JamesCheng]]

The other things that I noticed is that we create 900+ decoder threads(which means 900+ decoder) in the same time.

I don't know if this situation is caused by the fennec user with some abnormal manipulation or there are some bugs in gecko.

I will try to dig out if there is any code defect that will repeatedly create decoder.

BTW, since we introduce OOP decoding architecture in bug 1257777, the fennec will not crash logically...

So maybe this is the alternative solution for this bug.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

You need to request permission to view the hidden crash report fields if you don't have it already. The logcat is privacy sensitive just like URLs so you need special permission to access it. If you have permissions on your account, make sure you are logged in to socorro. I can also pull out the full logcat dump from that crashreport for you if you'd like.

Note that not all the crashes will have logcat.

Comment 10

[(inactive) Jim Chen [:jchen] [:darchons]]

A lot of the logs indicate OOM, so maybe we're creating too many decoder threads or forgot to terminate them. Can we limit to one or a few decoder threads?

Comment 11

[James Cheng[:JamesCheng]]

Hi kats,

Referring to the permission, where can I apply for the special permission?

Thanks.

Comment 12

[Jean-Yves Avenard [:jya]]

(In reply to Jim Chen [:jchen] [:darchons] from comment #10)
> A lot of the logs indicate OOM, so maybe we're creating too many decoder
> threads or forgot to terminate them. Can we limit to one or a few decoder
> threads?

There's already a mechanism in place, introduced in bug 1214710. This is controlled by a pref media.decoder.limit

But in any case, if the issue is with decoder not being freed, this is what needs to be fixed and not paper around the problem by simply limiting how many we create at once.

OUt of Process decoding is currently set for Windows decoder..

Comment 13

[Kartikaya Gupta (email:[e-mail])]

(In reply to James Cheng[:JamesCheng] from comment #11)
> Hi kats,
> 
> Referring to the permission, where can I apply for the special permission?
> 
> Thanks.

I got it back in the day by filing a bug - see https://bugzilla.mozilla.org/show_bug.cgi?id=818898. I'm not sure how one goes about it today.

Comment 14

[Blake Wu [:bwu][:blakewu]]

(In reply to Jean-Yves Avenard [:jya] from comment #12)
> OUt of Process decoding is currently set for Windows decoder..
We are using this mechanism on Fennec as well. Bug 1257777 is about it.

Comment 15

[James Cheng[:JamesCheng]]

Hi jya,

I saw the comment 12 you indicated that we have some mechanism to restrict the number of decoder creation.

For [1], does it means there are no decoder limit on all platform except Fennec which is under API Level 18?

Because I can only found the decoder is created numerous by crash report. 

Should we need to file another bug to limit the number of decoder creation on Fennec even API Level is higher than 18?
[1]
http://searchfox.org/mozilla-central/rev/d96317a351af8aa78ab9847e7feed964bbaac7d7/dom/media/MediaPrefs.h#175

Comment 16

[Jean-Yves Avenard [:jya]]

(In reply to James Cheng[:JamesCheng] from comment #15)
> Hi jya,
> 
> I saw the comment 12 you indicated that we have some mechanism to restrict
> the number of decoder creation.
> 
> For [1], does it means there are no decoder limit on all platform except
> Fennec which is under API Level 18?

the limit was set for < 18 because simply creating more than 2 MP3 decoders would cause a crash unrelated to OOM.

However, if the issue is due to OOM and the non-handling of those, it seems to me that limiting the number of decoders open at any one time is just papering over the real problem. It's not a fix as such.

Bug 1311872 would be a better work around than limiting the number of live decoders that is defined by platforms.

Comment 17

[James Cheng[:JamesCheng]]

The symptom of this bug is that we create lots of decoder and crash with OOM. Since Bug 1311872 is landing, I think the OOM symptom will be eased.

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

[Tracking Requested - why for this release]: Too late to take a fix for this in 50. Nom'ing for tracking in 51.

Comment 19

[Gerry Chang [:gchang]]

Track 51- as there are no crashes in 51 for now.

Comment 20

[Blake Wu [:bwu][:blakewu]]

Per comment 19.