1150918 - [Crash] [@ mozilla::MediaStreamGraphImpl::UpdateStreamOrder]

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Stability, defect)

Description

[Nicholas Troast [:ntroast]]

We observed the following crash signature during testing.

[@ mozilla::MediaStreamGraphImpl::UpdateStreamOrder | mozilla::MediaStreamGraphImpl::UpdateGraph | mozilla::MediaStreamGraphImpl::OneIteration | mozilla::AudioCallbackDriver::DataCallback ]

Cafbot will upload the decoded minidump and extra file.

Comment 1

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment -

Comment 2

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump -

Comment 3

[bhavana bajaj [:bajaj]]

roc/ehsan, any ideas here? See some changes from your side on m-c recently, but not sure if they are related. Can you please help investigate this crasher ?

Comment 4

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119
Moz BuildID: 20150330002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=473cd63f53c855299b719285d9b95e3f2910782f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=e44e28d2a37258fe0c908c59f2e91e5287777b40
Patches: bug 1133398, bug 1143694, bug 1146987, bug 1145724, bug 1147646, bug 1133147, bug 1150271, bug 1142770

Comment 5

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119

Comment 6

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.119

Comment 7

[(no longer active)]

Paul hacks on this stuff nowadays.

Comment 8

[Bobby Chien]

Steve, could you help investigate as well?

Comment 9

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.124
Moz BuildID: 20150406002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.124.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=a6351e1197d54f8624523c2db9ba1418f2aa046f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=6bb2afcce9872a7cbc65b4a58f752e2d5ac02345
Patches: bug 1145724, bug 1143694, bug 1146987, bug 1133398, bug 1133147, bug 1150271

Comment 10

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.124

Comment 11

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.124

Comment 12

[Paul Adenot (:padenot)]

Are there specific step to reproduce? It's crashing here [0].

[0]: http://git.mozilla.org/?p=releases/gecko.git;a=blob;f=dom/media/MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a;hb=522b546b5d6b8a91da23f0c52e97b85c3e7f3feb#l657

Comment 13

[Inder]

this crash was produced during stability tests which involves monkey testing for several hours and there is no clear STR for this. If we are not able to identify the issue using provided logs then please feel free to provide us a debug patch with additional logging to identify the issue.

Comment 14

[Robert O'Callahan (:roc) (email my personal email if necessary)]

This seems like it could be related to bug 1150923. Both bugs suggest corruption of the graph, so that a destroyed stream is still present as the source or destination of a MediaInputPort. But I don't remember seeing any bugs like that for a long time...

The MediaStream destructor logging patch I just attached to bug 1150923 might help here too. We should also add assertions that the mInputs and mConsumers arrays are empty when we destroy a MediaStream, to ensure that it's fully disconnected. Added here:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4725d387fbbe
https://hg.mozilla.org/try/rev/f15873e99ce5

Comment 15

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123
Moz BuildID: 20150405002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=a6351e1197d54f8624523c2db9ba1418f2aa046f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=6bb2afcce9872a7cbc65b4a58f752e2d5ac02345
Patches: bug 1145724, bug 1143694, bug 1146987, bug 1133398, bug 1152095, bug 1150924, bug 1133147, bug 1150271, bug 1150916

Comment 16

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123

Comment 17

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123

Comment 18

[StevenLee[:slee]]

(In reply to cafbot (PoC: ggrisco) from comment #17)
> Created attachment 8589618 [details]
> decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.123

From the log, I found that on frame 7 of Thread 20, it calls NS_ABORT_OOM. It looks like SetLength on [1] requests too many memory.

roc, paul, 
Do you have any thoughts about this?

[1] http://git.mozilla.org/?p=integration/gecko-dev.git;a=blob;f=dom/media/MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a;hb=643f37b68305d90ecfa148f81db63891c30e4620#l373

Comment 19

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to StevenLee[:slee] from comment #18)
> From the log, I found that on frame 7 of Thread 20, it calls NS_ABORT_OOM.
> It looks like SetLength on [1] requests too many memory.
> 
> roc, paul, 
> Do you have any thoughts about this?
> 
> [1]
> http://git.mozilla.org/?p=integration/gecko-dev.git;a=blob;f=dom/media/
> MediaStreamGraph.cpp;h=4a50ea3185be2f8f01ea44d2e52444e119c2e11a;
> hb=643f37b68305d90ecfa148f81db63891c30e4620#l373

In this log, there are two MSGs running, one driven by a ThreadedDriver (system clock or offline, unfortunately I can't tell which), and another driven by the audio callback. That's unusual, but possible if there's an OfflineAudioContext (but should not happen for any other reason).

Comment 20

[Bobby Chien]

Inder, could you help to provide more information? more log or reproduce steps will be helpful. thanks.

Comment 21

[Nicholas Troast [:ntroast]]

Looking at the provided logs I see that Ringtones app may be the culprit.
I am starting to see Ringtones as a recurring theme for these MSG issues.

I do have additional logs for this issue, but please be specific as they are very large.

Comment 22

[cafbot (PoC: ggrisco)]

Observed on: 

Device: msm8909
Gonk Version: AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.126
Moz BuildID: 20150406002503
Manifest: https://www.codeaurora.org/cgit/quic/lf/b2g/manifest/tree/caf_AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.126.xml?h=release
Gecko Version: 37.0
Gaia:  http://git.mozilla.org/?p=releases/gaia.git;a=commit;h=a6351e1197d54f8624523c2db9ba1418f2aa046f
Gecko: http://git.mozilla.org/?p=releases/gecko.git;a=commit;h=6bb2afcce9872a7cbc65b4a58f752e2d5ac02345
Patches: bug 1152361, bug 1150923, bug 1152095, bug 1150924, bug 1133147, bug 1150271, bug 1150916, bug 1148641

Comment 23

[cafbot (PoC: ggrisco)]

Attachment: EXTRA file attachment - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.126

Comment 24

[cafbot (PoC: ggrisco)]

Attachment: decoded minidump - AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.126

Comment 25

[Greg Grisco]

Not seen in past few builds, so closing this for now.