846465 - crash in mozilla::MediaPluginHost::CreateDecoder @ libstagefright.so@0x79877 when playing mp3 files on HTC devices running JB

Status: VERIFIED FIXED

(Core :: Audio/Video, defect)

Description

[Scoobidiver (away)]

It's #2 top crasher in 20.0b1, 21.0a2 and 22.0a1.

Here are affected devices:
*20.0:
HTC One X 	88
HTC One S 	22
HTC EVO 	10
HTC One X+ 	9
HTC6435LVW 	6
HTC J Z321e 	3
HTC One XL 	3
HTC Butterfly 	2
Rockchip TPC97809 	1
HTC X720d 	1
HUAWEI U9510E 	1
*21.0:
HTC One X 	27
HTC EVO 	3
HUAWEI U9508 	1
HTC6435LVW 	1
*22.0:
HTC One X 	24
HTC One S 	6
HTC One X+ 	3

Reports at:
https://crash-stats.mozilla.com/report/list?product=FennecAndroid&signature=%400x0

Comment 1

[Scoobidiver (away)]

It first showed up in 21.0a1/20121207. It's discontinuous across builds but there are crashes at least every three builds, so the regression range might be:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6fa6e55a93b2&tochange=739f20de3c1e

Comment 2

[Aaron Train [:aaronmt]]

Crashing via https://addons.mozilla.org/en-US/android/addon/flash-video-downloader-youtube ?

Comment 3

[Aaron Train [:aaronmt]]

Kats mentioned in channel that log-output from the reports was coming from https://mxr.mozilla.org/addons/source/6584/chrome/

Comment 4

[Kartikaya Gupta (email:[e-mail])]

I saw log output from that addon in one of the two crash report logcats I looked at; it may be unrelated.

Comment 5

[Scoobidiver (away)]

Most crash reports have no installed extensions.

Comment 6

[(inactive) Jim Chen [:jchen] [:darchons]]

I guess crash reporter is not seeing it, but there is a reasonable stack in the raw crash dump. Here is a stack trace from the report at https://crash-stats.mozilla.com/report/index/278e3a55-ab1d-4d41-b5a2-691622130227. I looked at stacks from several reports across different devices, and they all look more or less the same.

> 0x000: pc = 0x0
> 0x000: lr = libstagefright.so + 0x79877
> 0x470: libstagefright.so + 0x7a194
> 0x484: libstagefright.so + 0x7a1ba
> 0x48c: libstagefright.so + 0xca7dc
> 0x4b8: libstagefright.so + 0x7a194
> 0x4c4: libstagefright.so + 0x72eea
> 0x4f4: libstagefright.so + 0x8ab4a (MediaExtractor::Create)
> 0x534: libomxplugin.so + 0x1dae (OmxPlugin::OmxDecoder::Init+0x56)
> 0x5ec: libomxplugin.so + 0x23de (OmxPlugin::CreateDecoder+0x1a)
> 0x5fc: libxul.so + 0x70982e (mozilla::MediaPluginHost::CreateDecoder+0x42)
> 0x624: libxul.so + 0x709be4 (mozilla::MediaPluginReader::ReadMetadata+0x28)
> 0x694: libxul.so + 0x6f5d9a (mozilla::MediaDecoderStateMachine::DecodeMetadata+0x46)
> 0x6fc: libxul.so + 0x6f81d8 (mozilla::MediaDecoderStateMachine::DecodeThreadRun+0x24)
> 0x70c: libxul.so + 0x2b7238 (nsRunnableMethodImpl::Run+0x18)
> 0x714: libxul.so + 0xab0b18 (nsThread::ProcessNextEvent+0x184)
> 0x754: libxul.so + 0xa91d48 (NS_ProcessNextEvent_P+0x14)
> 0x764: libxul.so + 0xab08f8 (nsThread::ThreadFunc+0x50)

Maybe HTC introduced some more incompatibilities in their stagefright library?

Comment 7

[Scoobidiver (away)]

(In reply to Jim Chen [:jchen :nchen] from comment #6)
> Maybe HTC introduced some more incompatibilities in their stagefright
> library?
Why Firefox 19 is unaffected?

Comment 8

[Scoobidiver (away)]

Bug 818291 is in the regression range.

Comment 9

[(no longer active)]

(In reply to Scoobidiver from comment #8)
> Bug 818291 is in the regression range.

I have no idea how that patch could have caused this...

Comment 11

[Todd]

Bug 846892 is a duplicate and can be replicated by selecting a direct mp3 link for playback. The file shown in the screenshot is: http://traffic.libsyn.com/prestonandsteve/PnS022813.mp3 from http://prestonandsteve.libsyn.com/

Device: AT&T HTC One X+ running stock carrier Android rom 4.1.1

Comment 12

[Alex Keybl [:akeybl]]

Let's find a more specific regression range given the STR in comment 11.

Comment 13

[Aaron Train [:aaronmt]]

Mobile QA doesn't have any HTC OneX+ devices or variants. Todd, would you be willing to help us out by trying older mozilla-central-android (Nightly) builds from last December to narrow down the last known good day and the first known bad day? http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/12/

Comment 14

[Aaron Train [:aaronmt]]

cdouble, I think you have a One X according to (https://bugzilla.mozilla.org/show_bug.cgi?id=759945#c1)

Comment 15

[Andreea Pod]

I borrowed a phone outside the QA team and I found the regression window for this issue:

good build: 01.02.2012  
bad build:  02.02.2012 

possible pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d2fbc67f69f5&tochange=0352a32fde64

Comment 16

[Scoobidiver (away)]

Based on the STR about playing mp3 audio files, I suspect bug 455165 in that regression window.

Comment 17

[Paul Adenot (:padenot)]

Nope, bug 455165 has nothing to do with mp3 (it is an ogg-only patch). Not saying that bug 455165 is not responsible, but there is things that are more likely to be the cause, like bug 805700 and bug 787228.

Comment 18

[Scoobidiver (away)]

(In reply to Paul Adenot (:padenot) from comment #17)
> there is things that are more likely to be the cause, like bug 805700 and bug 787228.
Theoretically, those bugs are for different Android versions. Nevertheless, they might have side effects on Jelly Bean.

Comment 19

[cajbir (:cajbir)]

(In reply to Aaron Train [:aaronmt] from comment #14)
> cdouble, I think you have a One X according to
> (https://bugzilla.mozilla.org/show_bug.cgi?id=759945#c1)

I don't, but Edwin and Nick Cameron does.

Comment 20

[Scoobidiver (away)]

StageFright decoding is disabled by default in 20.0 for Gingerbread and Honeycomb, so backing out bug 805700 and bug 787228 from Beta wouldn't affect features and might fix those crashes.

Comment 21

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

Will try to reproduce this tonight.

Comment 22

[cajbir (:cajbir)]

(In reply to Scoobidiver from comment #20)
> StageFright decoding is disabled by default in 20.0 for Gingerbread and
> Honeycomb, so backing out bug 805700 and bug 787228 from Beta wouldn't
> affect features and might fix those crashes.

These bugs shouldn't affect JB devices. Is it possible the crashes have gotten more numerous due to wider adoption of the JB update on HTC devices rather than a change on our end?

Comment 23

[Scoobidiver (away)]

(In reply to Chris Double (:doublec) from comment #22)
> These bugs shouldn't affect JB devices.
Theoretically. Try to find other video bugs in the regression range of comment 15.

> Is it possible the crashes have gotten more numerous due to wider adoption of the JB
> update on HTC devices rather than a change on our end?
19.0 would have been affected.

Comment 24

[Aaron Train [:aaronmt]]

Confirming that I don't see this problem on a HTC Gingerbread device (Desire HD). I don't have any newer HTC devices.

Comment 25

[cajbir (:cajbir)]

(In reply to Edwin Flores [:eflores] [:edwin] from comment #21)
> Will try to reproduce this tonight.

Any luck?

Comment 26

[Scoobidiver (away)]

Here is the list of audio/video bugs in the regression range of comment 15: bug 455165, bug 816576, bug 787228, bug 805700, and bug 813562.
Paul, what about bug 816576?

Comment 27

[Paul Adenot (:padenot)]

Not bug 816576, because according to the stack track, at this point in the audio/video setup, haven't called yet into SoundTouch (that happens later, when the audio output is created).

Comment 28

[Aaron Train [:aaronmt]]

I'd encourage Chris/Edwin or Paul to order a device to progress the state of this bug since guessing change-sets in that window doesn't seem to be yielding anything.

Comment 29

[Ioana Chiorean]

(In reply to Andreea Pod from comment #15)
> I borrowed a phone outside the QA team and I found the regression window for
> this issue:
> 
> good build: 01.02.2012  
> bad build:  02.02.2012 
> 
> possible pushlog:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=d2fbc67f69f5&tochange=0352a32fde64

This was reproduced on HTC One S (JB). 

Not reproducing on HTC Desire (2.2)- we got the following message: "Video can't be played because the file is corrupt." without crashing.

Comment 30

[Scoobidiver (away)]

(In reply to Ioana Chiorean from comment #29)
> Not reproducing on HTC Desire (2.2)- we got the following message: "Video
> can't be played because the file is corrupt." without crashing.
It's bug 851261.

Comment 31

[Alex Keybl [:akeybl]]

Assigning to Edwin - we're now two weeks away from release. We'd like to avoid shipping a reproducible topcrash regression, so please make this a high priority.

Comment 32

[cajbir (:cajbir)]

(In reply to Aaron Train [:aaronmt] from comment #28)
> I'd encourage Chris/Edwin or Paul to order a device to progress the state of
> this bug since guessing change-sets in that window doesn't seem to be
> yielding anything.

We're waiting on Edwin responding to the needinfo. He has a device and once we can run some tests on it we should be able to resolve this pretty quickly if it is a regression.

Comment 33

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

Managed to reproduce on my HTC One X; even after backing out the stagefright patches from comment 15.

Comment 34

[Scoobidiver (away)]

(In reply to Edwin Flores [:eflores] [:edwin] from comment #33)
> Managed to reproduce on my HTC One X; even after backing out the stagefright
> patches from comment 15.
So the remaining suspected bugs are bug 455165, bug 816576, and bug 813562. Can you back them out one by one?

Comment 35

[cajbir (:cajbir)]

(In reply to Scoobidiver from comment #34)
> So the remaining suspected bugs are bug 455165, bug 816576, and bug 813562.
> Can you back them out one by one?

The crash looks to be caused by the DataSource class on the phone having a different layout to the standard stagefright class that we are expecting. If we use Android's "CreateFromURI" to hardcode a video, the DataSource we get back works but our own DataSource crashes. This seems to indicate the issue is not a regression but rather a change in the stagefright library on the device. I doubt Jellybean ever worked on the HTC One X or it has received an update that stopped it from working.

Comment 36

[Todd]

(In reply to Aaron Train [:aaronmt] from comment #13)
> Mobile QA doesn't have any HTC OneX+ devices or variants. Todd, would you be
> willing to help us out by trying older mozilla-central-android (Nightly)
> builds from last December to narrow down the last known good day and the
> first known bad day?
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/12/


The Nightly build from December 1st (http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/12/2012-12-01-03-08-12-mozilla-central-android/) gives an error message "Video can't be played because the file is corrupt" when attempting to play mp3 files from the website http://prestonandsteve.libsyn.com/ and podfeed.net

Attempting to play the same mp3 files in Nightly builds from December 2nd (http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2012/12/2012-12-02-03-07-23-mozilla-central-android/) and onward causes the browser to crash.

Device: AT&T HTC One X+ running stock carrier Android 4.1.1

Comment 37

[Scoobidiver (away)]

(In reply to Chris Double (:doublec) from comment #35)
> I doubt Jellybean ever worked on the HTC One X or it has received an update that 
> stopped it from working.
I has never worked (see bug 851261) but it didn't crash in 19.0.2 up to 20.0a1/20121201.

Comment 38

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

Attachment: Fixy Fix

Comment 39

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

Attachment: Fixy Fix

Fixed grammar badness in comment.

Comment 40

[cajbir (:cajbir)]

Comment on attachment 727501 [details] [diff] [review]
Fixy Fix

Review of attachment 727501 [details] [diff] [review]:
-----------------------------------------------------------------

r+ for the media changes with miner string fix below. You'll need to get relevant reviews for the other files. I suggest mark.finkle for package-manifest.in and a build peer for packager.mk, toolkit.mozbuild.

::: content/media/plugins/MediaPluginHost.cpp
@@ +166,5 @@
>  
> +  nsAutoString manufacturer;
> +  rv = infoService->GetPropertyAsAString(NS_LITERAL_STRING("manufacturer"), manufacturer);
> +  if (NS_SUCCEEDED(rv)) {
> +    ALOG("Android Device is: %s", NS_LossyConvertUTF16toASCII(manufacturer).get());

That should say "Android Manufacturer is"

Comment 41

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 727501 [details] [diff] [review]
Fixy Fix

package files are OK

Comment 43

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1cb7fcc85525

Comment 44

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/1cb7fcc85525

Comment 45

[Robert Kaiser]

As this is the #2 topcrash on 20.0b5 for Android, do we still want to get this uplifted to Beta? If not, I think we want Aurora at least.

Comment 46

[Scoobidiver (away)]

(In reply to Robert Kaiser (:[e-mail]) from comment #45)
> As this is the #2 topcrash on 20.0b5 for Android, do we still want to get
> this uplifted to Beta?
As long as release drivers haven't set status-firefox20 to wontfix, it's worth requesting an uplift to Beta with the usual risk assessment.

Comment 47

[Lukas Blakk [:lsblakk] use ?needinfo]

This would be great to uplift if it's low risk to take in our final beta (minimal testing/verification window) - Chris or Mark can you make the case about the risk here?

Comment 48

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

Comment on attachment 727501 [details] [diff] [review]
Fixy Fix

[Approval Request Comment]
Bug caused by (feature/regressing bug #):
libstagefright
User impact if declined:
Crashing on mp3 and mp4 content on JB HTC devices
Testing completed (on m-c, etc.):
OK in Nightly
Risk to taking this patch (and alternatives if risky):
Worst case, crashy device continues to crash.
String or UUID changes made by this patch:
None

Comment 49

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 727501 [details] [diff] [review]
Fixy Fix

Approving for uplift since we want to try and reduce crashes on these specific devices and the risk to anything else seems negligible so it's worth a shot. Please land to branches asap.

Comment 50

[Lukas Blakk [:lsblakk] use ?needinfo]

For relnote we'll want to point to https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers#On_Android_2 (also in Google Play "what's new")

Comment 51

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/b78189da9a23
https://hg.mozilla.org/releases/mozilla-beta/rev/5573b93cd43d

Comment 52

[Catalin Suciu [:csuciu]]

On FF 20 Beta 7 & Aurora 21.0a2 (2013-03-26):
 - no longer crashes when trying to play an .mp3 file from http://prestonandsteve.libsyn.com/ but the file is not played - the player is not created

On Nightly 22.0a1 (2013-03-26) 
 - no crash and the mp3's from http://prestonandsteve.libsyn.com/ are played correctly
 
Device: HTC One S (Android 4.1.1)