Intermittent test_browserElement_oop_AudioChannel_nested.html | This test left crash dumps behind, but we weren't expecting it to!

RESOLVED DUPLICATE of bug 1239873

Status

()

P1
normal
Rank:
15
RESOLVED DUPLICATE of bug 1239873
3 years ago
3 years ago

People

(Reporter: hiro, Assigned: jib)

Tracking

({intermittent-failure})

Other Branch
intermittent-failure
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

https://treeherder.mozilla.org/logviewer.html#?job_id=14975049&repo=try

The crash dumps:
17:52:34     INFO -  #01: mozilla::MediaStreamGraphImpl::EnsureStableStateEventPosted() [dom/media/MediaStreamGraph.cpp:1592]
17:52:34     INFO -  #02: mozilla::MediaStreamGraphImpl::PrepareUpdatesToMainThreadState(bool) [dom/media/MediaStreamGraph.cpp:972]
17:52:34     INFO -  #03: mozilla::MediaStreamGraphImpl::UpdateMainThreadState() [dom/media/MediaStreamGraph.cpp:1230]
17:52:34     INFO -  #04: mozilla::MediaStreamGraphImpl::OneIteration(__int64) [dom/media/MediaStreamGraph.cpp:1271]
17:52:34     INFO -  #05: mozilla::AudioCallbackDriver::DataCallback(float *,long) [dom/media/GraphDriver.cpp:880]
17:52:34     INFO -  #06: mozilla::AudioCallbackDriver::DataCallback_s(cubeb_stream *,void *,void *,long) [dom/media/GraphDriver.cpp:731]
17:52:34     INFO -  #07: noop_resampler::fill(void *,long) [media/libcubeb/src/cubeb_resampler.cpp:89]
17:52:34     INFO -  #08: cubeb_resampler_fill [media/libcubeb/src/cubeb_resampler.cpp:245]
17:52:34     INFO -  #09: `anonymous namespace'::refill(cubeb_stream *,float *,long) [media/libcubeb/src/cubeb_wasapi.cpp:475]
17:52:34     INFO -  #10: `anonymous namespace'::wasapi_stream_render_loop [media/libcubeb/src/cubeb_wasapi.cpp:604]
17:52:34     INFO -  #11: MSVCR120 + 0x2c01d
17:52:34     INFO -  #12: MSVCR120 + 0x2c001
17:52:34     INFO -  #13: kernel32 + 0x53c45
17:52:34     INFO -  #14: ntdll + 0x637f5
17:52:34     INFO -  #15: ntdll + 0x637c8
17:52:34     INFO -  [Child 1636] WARNING: '!mMainThread', file c:/builds/moz2_slave/try-w32-d-00000000000000000000/build/src/xpcom/threads/nsThreadManager.cpp, line 295
17:52:34     INFO -  [Child 1636] WARNING: 'NS_FAILED(rv)', file c:/builds/moz2_slave/try-w32-d-00000000000000000000/build/src/xpcom/glue/nsThreadUtils.cpp, line 191
17:52:34     INFO -  [Child 1636] ###!!! ASSERTION: Failed NS_DispatchToMainThread() in shutdown; leaking: 'false', file c:/builds/moz2_slave/try-w32-d-00000000000000000000/build/src/xpcom/glue/nsThreadUtils.cpp, line 192
17:52:34     INFO -  #01: mozilla::MediaStreamGraphImpl::RunInStableState(bool) [dom/media/MediaStreamGraph.cpp:1534]
17:52:34     INFO -  #02: mozilla::`anonymous namespace'::MediaStreamGraphStableStateRunnable::Run() [dom/media/MediaStreamGraph.cpp:1373]
17:52:34     INFO -  #03: mozilla::CycleCollectedJSRuntime::ProcessStableStateQueue() [xpcom/base/CycleCollectedJSRuntime.cpp:1048]
17:52:34     INFO -  #04: mozilla::CycleCollectedJSRuntime::~CycleCollectedJSRuntime() [xpcom/base/CycleCollectedJSRuntime.cpp:460]
17:52:34     INFO -  #05: nsXPConnect::~nsXPConnect() [js/xpconnect/src/nsXPConnect.cpp:101]
17:52:34     INFO -  #06: nsXPConnect::ReleaseXPConnectSingleton() [js/xpconnect/src/nsXPConnect.cpp:149]
17:52:34     INFO -  #07: nsTArray_Impl<nsAutoPtr<nsComponentManagerImpl::KnownModule>,nsTArrayInfallibleAllocator>::RemoveElementsAt(unsigned int,unsigned int) [xpcom/glue/nsTArray.h:1655]
17:52:34     INFO -  #08: nsComponentManagerImpl::Shutdown() [xpcom/components/nsComponentManager.cpp:927]
17:52:34     INFO -  #09: XRE_TermEmbedding [toolkit/xre/nsEmbedFunctions.cpp:210]
17:52:34     INFO -  #10: content_process_main(int,char * * const) [ipc/contentproc/plugin-container.cpp:238]
17:52:34     INFO -  #11: wmain [toolkit/xre/nsWindowsWMain.cpp:138]
17:52:34     INFO -  #12: __tmainCRTStartup [f:/dd/vctools/crt/crtw32/startup/crt0.c:255]
17:52:34     INFO -  #13: kernel32 + 0x53c45
17:52:34     INFO -  #14: ntdll + 0x637f5
17:52:34     INFO -  #15: ntdll + 0x637c8
Duplicate of this bug: 1235972
I hit the 0xe5e5e5e5 crash from the duped bug on a Try push yesterday too.
Component: Audio/Video → Audio/Video: MediaStreamGraph
This was deep in shutdown (after final-CC), which also implies something about the shutdown sequence.  The others duped to it don't appear to be, but that's less certain
Rank: 15
Priority: -- → P1
9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 3
* mozilla-central: 2
* b2g-inbound: 2
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* windows7-32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2015-12-28&endday=2016-01-03&tree=all
30 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 18
* fx-team: 7
* mozilla-central: 3
* b2g-inbound: 2

Platform breakdown:
* windows8-64: 20
* windows7-32: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-06&endday=2016-01-06&tree=all
FYI: Padenot is looking at this now.  Once we know more, I'll assign an owner.
37 automation job failures were associated with this bug yesterday.

Repository breakdown:
* fx-team: 15
* mozilla-inbound: 14
* b2g-inbound: 5
* mozilla-central: 2
* try: 1

Platform breakdown:
* windows8-64: 26
* windows7-32: 8
* mulet-linux64: 2
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-07&endday=2016-01-07&tree=all
37 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 8
* try: 4
* mozilla-central: 4
* b2g-inbound: 4

Platform breakdown:
* windows8-64: 21
* windows7-32: 15
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-08&endday=2016-01-08&tree=all
17 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 7
* mozilla-central: 5
* fx-team: 4
* b2g-inbound: 1

Platform breakdown:
* windows8-64: 10
* windows7-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-09&endday=2016-01-09&tree=all
145 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 68
* fx-team: 37
* mozilla-central: 14
* b2g-inbound: 14
* try: 12

Platform breakdown:
* windows8-64: 86
* windows7-32: 55
* mulet-linux64: 2
* linux64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-04&endday=2016-01-10&tree=all
This is likely yet another racy shutdown issue with MSG; note the failure to DispatchToMainThread right before.

We should use the AysncShutdown blockers to hold shutdown at xpcom-shutdown (and/or xpcom-shutdown-threads) until MSG has shutdown the callback driver or MSG thread, and processed all messages (and blocked new ones), then release the shutdown process.  Likely this will *greatly* simplify the shutdown code in MSG and remove any chance of races.

MediaManager is using this now; see bug 1166293 and bug 1237794

NI-ing roc since he had a big hand in the current shutdown sequence for MSG for any comments/suggestions
Flags: needinfo?(roc)
21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* fx-team: 4
* try: 3
* b2g-inbound: 3
* mozilla-central: 2

Platform breakdown:
* windows7-32: 15
* windows8-64: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-11&endday=2016-01-11&tree=all
(In reply to Randell Jesup [:jesup] from comment #11)
> This is likely yet another racy shutdown issue with MSG; note the failure to
> DispatchToMainThread right before.
> 
> We should use the AysncShutdown blockers to hold shutdown at xpcom-shutdown
> (and/or xpcom-shutdown-threads) until MSG has shutdown the callback driver
> or MSG thread, and processed all messages (and blocked new ones), then
> release the shutdown process.  Likely this will *greatly* simplify the
> shutdown code in MSG and remove any chance of races.
> 
> MediaManager is using this now; see bug 1166293 and bug 1237794
> 
> NI-ing roc since he had a big hand in the current shutdown sequence for MSG
> for any comments/suggestions

Yes, it sounds like that would help.
Flags: needinfo?(roc)
Jan-Ivar is going to rework the shutdown code for MediaStreamGraph -- much like he did for MediaManager (where he used AsyncShutdown).  I'm assigning this bug to him, but the plan is for him to open a new bug to rework the MSG shutdown code to make it less fragile, which in the process should fix this intermittent.

If this intermittent gets too annoying before we can land the new shutdown code, let me know.  I've asked jib to work on this next, after bug 1232082 lands and sticks.
Assignee: nobody → jib
Depends on: 1239873

Updated

3 years ago
See Also: → bug 1238542
43 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 24
* fx-team: 10
* try: 3
* mozilla-central: 3
* b2g-inbound: 3

Platform breakdown:
* windows7-32: 28
* windows8-64: 14
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1235968&startday=2016-01-11&endday=2016-01-17&tree=all
Oddly, it seems these subsided around Jan 14, predating bug 1239873 by several days, so there might be a different source for this fix. I'm closing it still based on comment 11 however.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1239873
This type of problem was very susceptible to shutdown races, so it can come and go.
You need to log in before you can comment on or make changes to this bug.