1274797 - Crash in OOM | large | mozalloc_abort | mozalloc_handle_oom | moz_xrealloc | nsTArray_Impl<T>::Clear | nsTArray_base<T>::EnsureCapacity<T> | mozilla::MediaStreamGraphImpl::PrepareUpdatesToMainThreadState

Status: RESOLVED DUPLICATE of bug 1275754

(Core :: Audio/Video: MediaStreamGraph, defect, P1)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-ec6abc1a-b1a4-4388-a0b2-6eac02160521.
=============================================================
Crashing Thread (54)
Frame 	Module 	Signature 	Source
0 	mozglue.dll 	mozalloc_abort(char const* const) 	memory/mozalloc/mozalloc_abort.cpp:33
1 	mozglue.dll 	mozalloc_handle_oom(unsigned int) 	memory/mozalloc/mozalloc_oom.cpp:46
2 	mozglue.dll 	moz_xrealloc 	memory/mozalloc/mozalloc.cpp:107
3 	xul.dll 	nsTArray_Impl<ColorStop, nsTArrayInfallibleAllocator>::Clear() 	xpcom/glue/nsTArray.h:1673
4 	xul.dll 	nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator>(unsigned int, unsigned int) 	xpcom/glue/nsTArray-inl.h:183
5 	xul.dll 	mozilla::MediaStreamGraphImpl::PrepareUpdatesToMainThreadState(bool) 	dom/media/MediaStreamGraph.cpp:1081
6 	xul.dll 	mozilla::MediaStreamGraphImpl::UpdateMainThreadState() 	dom/media/MediaStreamGraph.cpp:1342
7 	xul.dll 	mozilla::MediaStreamGraphImpl::OneIteration(__int64) 	dom/media/MediaStreamGraph.cpp:1384
8 	xul.dll 	mozilla::AudioCallbackDriver::DataCallback(float const*, float*, long) 	dom/media/GraphDriver.cpp:916
9 	xul.dll 	mozilla::AudioCallbackDriver::DataCallback_s(cubeb_stream*, void*, void const*, void*, long) 	dom/media/GraphDriver.cpp:759
10 	xul.dll 	noop_resampler::fill(void*, void*, long) 	media/libcubeb/src/cubeb_resampler.cpp:92
11 	xul.dll 	`anonymous namespace'::refill(cubeb_stream*, float*, long) 	media/libcubeb/src/cubeb_wasapi.cpp:475
12 	xul.dll 	`anonymous namespace'::wasapi_stream_render_loop 	media/libcubeb/src/cubeb_wasapi.cpp:604
13 	msvcr120.dll 	_callthreadstartex 	f:\dd\vctools\crt\crtw32\startup\threadex.c:376
14 	msvcr120.dll 	msvcr120.dll@0x2c000 	
15 	kernel32.dll 	BaseThreadInitThunk 	
16 	ntdll.dll 	__RtlUserThreadStart 	
17 	ntdll.dll 	_RtlUserThreadStart

this crash signature on windows systems seems to be regressing since the firefox 43 release and looks related to the change at https://hg.mozilla.org/releases/mozilla-release/rev/61079e59ef35#l1.106

Comment 1

[Karl Tomlinson (:karlt)]

Yes, that does seem a likely trigger, thanks.

https://bugzilla.mozilla.org/show_bug.cgi?id=1189562#c4 also wants to undo that change.

Comment 2

[Maire Reavy [:mreavy]]

Given we're seeing 400+ crashes in the last week, I'd like to make this a P1.  Karl -- Do you have the time to work on this now (since you seem to know the most about it), or should I find someone else?  Thanks.

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

#47 topcrash on release, not really huge volume here but it is notable. 
Wontfix for 46 and 47 since we don't have anything to act on here, and we're 2 weeks away from 47 release. 

This isn't showing up yet on 48/49. But it may once 48 is in beta. Tracking request for 48 so we will be sure to check up on it.

Comment 4

[Karl Tomlinson (:karlt)]

I'll have a look to see what can be done quickly.

61079e59ef35 may make things worse, but there was already a problem.
https://crash-stats.mozilla.com/signature/?signature=OOM+%7C+large+%7C+mozalloc_abort+%7C+mozalloc_handle_oom+%7C+moz_xrealloc+%7C+nsTArray_base%3CT%3E%3A%3AEnsureCapacity+%7C+mozilla%3A%3AMediaStreamGraphImpl%3A%3APrepareUpdatesToMainThreadState
is similar and happening on older branches without 61079e59ef35, including 38
esr and 40.

OOMAllocationSize is typically 1, 2, 4, 8 MB, but is close to 100MB in
https://crash-stats.mozilla.com/report/index/431e8bc1-0d1d-4f92-b64a-e23572160523#allthreads

Each element in the array would be 16 bytes, so I suspect there are many more
updates than streams, as if the main thread is not emptying the queue.  The
first few reports I looked at supported that hypothesis, with the main thread
in GC or CC.

I think we can limit the number of updates in the queue by removing obsolete updates.  We should also be able to reduce the number of updates being sent, but I'm guessing that accumulation of obsolete updates is the core problem to fix here.

Comment 5

[Karl Tomlinson (:karlt)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #3)
> #47 topcrash on release, not really huge volume here but it is notable. 
> Wontfix for 46 and 47 since we don't have anything to act on here, and we're
> 2 weeks away from 47 release. 

Where do you see this as #47?

https://crash-stats.mozilla.com/topcrashers/?product=Firefox&version=46.0.1&days=7
has 1556 crashes for #50 in that version.

https://crash-stats.mozilla.com/search/?signature=~%3APrepareUpdatesToMainThreadState&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
has 894 crashes in all versions.
"by default, the date will be a range between now and a week ago"

> This isn't showing up yet on 48/49.

Interesting, thanks.

https://crash-stats.mozilla.com/signature/?signature=OOM+%7C+large+%7C+mozalloc_abort+%7C+mozalloc_handle_oom+%7C+moz_xrealloc+%7C+nsTArray_base%3CT%3E%3A%3AEnsureCapacity+%7C+nsTArray_Impl%3CT%3E%3A%3ASetCapacity+%7C+mozilla%3A%3AMediaStreamGraphImpl%3A%3APrepareUpdatesToMainThreadState
goes back to version 29.

I don't know the trend but any increase would be likely due to more use of the newer media features, guessing Web Audio.

This affects 32-bit platforms, x86/Windows and ARM/Android.

Comment 6

[Karl Tomlinson (:karlt)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6dba4534dba

Comment 7

[Karl Tomlinson (:karlt)]

https://crash-stats.mozilla.com/signature/?date=%3E%3D2016-03-07&release_channel=aurora&signature=OOM%20%7C%20large%20%7C%20mozalloc_abort%20%7C%20mozalloc_handle_oom%20%7C%20moz_xrealloc%20%7C%20nsTArray_base%3CT%3E%3A%3AEnsureCapacity%3CT%3E%20%7C%20mozilla%3A%3AMediaStreamGraphImpl%3A%3APrepareUpdatesToMainThreadState&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&page=1#reports
has 10 crashes on 48.0a2, but the most recent was 20160513004028.

On Nightly 49, there were crashes with 20160517030211 and 20160428030218
https://crash-stats.mozilla.com/report/index/fb603faa-bebd-4ec5-83e1-cb93b2160526 
https://crash-stats.mozilla.com/report/index/664457d2-da0e-4405-b8c6-278202160430

Bug 1275754 landed a fix for the growing queue problem suspected in comment 4.
Once that is on m-c, we can think about uplift to 48.

Comment 8

[Karl Tomlinson (:karlt)]

https://crash-stats.mozilla.com/report/index/6be5787a-6019-4a29-a353-73fc82160529
was with 49.0a1 20160527030220.

Comment 9

[Maire Reavy [:mreavy]]

Karl -- Bug 1275754 has landed and uplifted to Fx48 (which is great).  This bug is currently tracking Fx48.  What do we want to do in this bug for Fx48?  (i.e. What are the next steps for this bug?)

Comment 10

[Karl Tomlinson (:karlt)]

(In reply to Maire Reavy [:mreavy] (Plz ni?:mreavy) from comment #9)
> Karl -- Bug 1275754 has landed and uplifted to Fx48 (which is great).  This
> bug is currently tracking Fx48.  What do we want to do in this bug for Fx48?
> (i.e. What are the next steps for this bug?)

Thanks for the reminder.
The next steps were to check crash stats.

Still no crashes on aurora with more recent builds.

48 beta 1 is released tomorrow, so I think we should leave this open and check
back in a week.

Comment 11

[Karl Tomlinson (:karlt)]

Adding the signature for crashes affecting uses on versions 36-40 (mostly ESR).

Comment 12

[Karl Tomlinson (:karlt)]

In the last 7 days there have been no reports with PrepareUpdatesToMainThreadState in the signature from builds with bug 1275754 fixed.

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Fixed in the duplicate.