1295663 - Async shutdown crash with "MediaShutdownManager shutdown"

Status: RESOLVED WORKSFORME

(Core :: Audio/Video: Playback, defect, P1)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-10e0d536-b8e5-43aa-a8e0-35a822160816.
=============================================================

starting with 49 builds there is a new async shutdown timeout abort, indicating that something in MediaShutdownManager code is blocking shutdown for a minute: http://bit.ly/2aX1pAU

this is happening across all platforms and currently responsible for about a quarter of async shutdown timeout crashes on 49.0b (which is equal to ~0.6% of all browser crashes). 
those crashes are annotated with {"phase":"profile-before-change","conditions":[{"name":"MediaShutdownManager: shutdown","state":"(none)","filename":"c:/builds/moz2_slave/m-beta-w32-0000000000000000000/build/src/dom/media/MediaShutdownManager.cpp","lineNumber":75,"stack":"MediaShutdownManager shutdown"}]} or similar.

Comment 1

[Liz Henry (:lizzard) (relman/hg->git project)]

#4 crash signature in recent betas (well, the entire signature... but this class of crashes sounds like a good part of that). 
Anthony can you help find someone to look at this? Thanks!

Comment 2

[JW Wang [:jwwang] [:jw_wang]]

https://crash-stats.mozilla.com/report/index/10e0d536-b8e5-43aa-a8e0-35a822160816#allthreads

Thread 49 got stuck in mozilla::MFTDecoder::Output(RefPtr<IMFSample>*).

Hi Chris,
Do you have any idea about what might block it?

Comment 3

[David Bolter [:davidb] (NeedInfo me for attention)]

Is it worth getting a regression window here?

Comment 4

[[:philipp]]

the first nightly where this is showing up was 20160530071207 (after that it is pretty much recurring at least once within each build).

changelog from 20160530071207 -1 day: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=2c7440e46d8786b2c82a1d2004e2b6d9d13f4046&tochange=3435dd7ad71fe9003bdeee18fd38d815e033beef

changelog from 20160530071207 -3 days: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=4d63dde701b47b8661ab7990f197b6b60e543839&tochange=3435dd7ad71fe9003bdeee18fd38d815e033beef

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Bug that touched media code in that range: (1 day):  Bug 1207830  (seth),  Bug 1275538 (jya).
Of course the issue might be elsewhere and media code is just affected.   Regression-range within that might help, but that would require a reproducible case.  Worth looking at jw's find, and also the landed for the 3 day period

Comment 6

[Maire Reavy [:mreavy]]

Anthony -- Can you find an owner for this?  It's on relman's radar as a new regression that should have an owner.  Thanks.

Comment 7

[Maire Reavy [:mreavy]]

(In reply to Maire Reavy [:mreavy] from comment #6)
> Anthony -- Can you find an owner for this?  It's on relman's radar as a new
> regression that should have an owner.  Thanks.

Comment 8

[Chris Pearce [:cpearce (Not reading bugmail)]]

This looks like another variant on bug 1274498.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'Abort | mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | NS_InvokeByIndex':
 - nightly (version 51): 226 crashes from 2016-08-01.
 - aurora  (version 50): 507 crashes from 2016-08-01.
 - beta    (version 49): 9322 crashes from 2016-08-02.
 - release (version 48): 7216 crashes from 2016-07-25.
 - esr     (version 45): 2309 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly      70      59      55
 - aurora      208     142      37
 - beta       3237    3260    1001
 - release    2514    2004    1079
 - esr         243     251     254

Affected platforms: Windows, Linux

Crash rank on the last 7 days:
           Browser     Content   Plugin
 - nightly #8
 - aurora  #7
 - beta    #3
 - release #3
 - esr     #34

Comment 10

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to Maire Reavy [:mreavy] from comment #6)
> Anthony -- Can you find an owner for this?  It's on relman's radar as a new
> regression that should have an owner.  Thanks.

c8 suggests that it isn't a regression, rather a duplicate of a long standing head scratcher.

Comment 11

[Liz Henry (:lizzard) (relman/hg->git project)]

Confusing bug/signature that bleeds into many other bugs. If we come up with a fix, I'll leave it to jwwang and anthony's team to request uplift or mark as wontfix for 49.

Comment 12

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Chris Pearce (:cpearce) from comment #8)
> This looks like another variant on bug 1274498.

The crash in comment #1 is an instance of bug 1274498. The crashes listed under the crash signature that :philipp added on 1 September is different from the crash in comment #1, and this isn't a variant of bug 1274498, it's something else.

Comment 13

[Chris Pearce [:cpearce (Not reading bugmail)]]

There are a variety of crashes coming in under the "AsyncShutdownTimeout|profile-before-change|MediaShutdownManager:shutdown" signature.

Most are instances of bug 1274498, which is fixed in 51, and has uplift requested.

There are some instances of some with threads inside libcubeb's setup_wasapi_stream_one_side<IAudioRenderClient>. For example:
https://crash-stats.mozilla.com/report/index/ca39d06f-7400-4a1d-b9bc-530662160909#allthreads
This is bug 1135562.

Another has threads shutting down cubeb, such has:
https://crash-stats.mozilla.com/report/index/a0142abb-b086-4f18-91f0-5ebf52160909#allthreads
This is bug 1274479.

One looks like a hang in the OSX decoder:
https://crash-stats.mozilla.com/report/index/bdacd7d8-9da4-4921-8faa-83fc82160908#allthreads
I've only seen the one of those.

So the most common case of these is fixed, and the other two cases we have bugs on file.

Comment 14

[Chris Pearce [:cpearce (Not reading bugmail)]]

The [@ Abort | mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | NS_InvokeByIndex] signature seems to only affect Firefox 29 (except for 1 instance of an unknown hang) so I'm removing it from this bug.

Comment 15

[Chris Pearce [:cpearce (Not reading bugmail)]]

Going to resolve this, since bug 1274498 is fixed, and other bugs are on file for the other things that got lumped in with this crash.

Comment 16

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Copying status over from bug 1274498.

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

Crash volume for signature 'AsyncShutdownTimeout | profile-before-change | MediaShutdownManager: shutdown':
 - nightly (version 52): 17 crashes from 2016-09-19.
 - aurora  (version 51): 15 crashes from 2016-09-19.
 - beta    (version 50): 869 crashes from 2016-09-20.
 - release (version 49): 4957 crashes from 2016-09-05.
 - esr     (version 45): 0 crashes from 2016-06-01.

Crash volume on the last weeks (Week N is from 10-03 to 10-09):
            W. N-1  W. N-2
 - nightly      11       6
 - aurora       13       2
 - beta        701     168
 - release    3994     962
 - esr           0       0

Affected platforms: Windows, Mac OS X, Linux

Crash rank on the last 7 days:
           Browser     Content   Plugin
 - nightly #79
 - aurora  #78
 - beta    #16
 - release #5
 - esr

Comment 18

[Marco Castelluccio [:marco]]

Given that bug 1274498 is wontfix for 49 and that we have closed this bug
in favor of bug 1274498, I guess we can set wontfix for 49.