1468660 - Crash in OOM | large | mozalloc_abort | mozalloc_handle_oom | moz_xrealloc | nsTArray_base<T>::EnsureCapacity<T> | nsTArray_Impl<T>::AppendElement<T> | mozilla::DecoderDoctorDocumentWatcher::AddDiagnostics

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is
report bp-6b407453-8a9f-4602-8e5c-3e68e0180530.
=============================================================

Top 10 frames of crashing thread:

0 mozglue.dll mozalloc_abort memory/mozalloc/mozalloc_abort.cpp:34
1 mozglue.dll mozalloc_handle_oom memory/mozalloc/mozalloc_oom.cpp:54
2 mozglue.dll moz_xrealloc memory/mozalloc/mozalloc.cpp:95
3 xul.dll nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::EnsureCapacity<nsTArrayInfallibleAllocator> xpcom/ds/nsTArray-inl.h:183
4 xul.dll nsTArray_Impl<mozilla::DecoderDoctorDocumentWatcher::Diagnostics, nsTArrayInfallibleAllocator>::AppendElement<mozilla::DecoderDoctorDocumentWatcher::Diagnostics, nsTArrayInfallibleAllocator> xpcom/ds/nsTArray.h:2286
5 xul.dll mozilla::DecoderDoctorDocumentWatcher::AddDiagnostics dom/media/doctor/DecoderDoctorDiagnostics.cpp:778
6 xul.dll mozilla::DecoderDoctorDiagnostics::StoreFormatDiagnostics dom/media/doctor/DecoderDoctorDiagnostics.cpp:859
7 xul.dll mozilla::dom::HTMLMediaElement::CanPlayType dom/html/HTMLMediaElement.cpp:4719
8 xul.dll static bool mozilla::dom::HTMLMediaElementBinding::canPlayType dom/bindings/HTMLMediaElementBinding.cpp:430
9 xul.dll js::ConcatStrings<1> js/src/vm/StringType.cpp:699

=============================================================

this oom signature on windows is mainly hitting 32bit users on windows - recorded oom allocation sizes can go up to close to 1GB.

the signature seems to spike up at various times for particular channels. for example:
*esr: https://crash-stats.mozilla.com/signature/?product=Firefox&release_channel=esr&signature=OOM%20%7C%20large%20%7C%20mozalloc_abort%20%7C%20mozalloc_handle_oom%20%7C%20moz_xrealloc%20%7C%20nsTArray_base%3CT%3E%3A%3AEnsureCapacity%3CT%3E%20%7C%20nsTArray_Impl%3CT%3E%3A%3AAppendElement%3CT%3E%20%7C%20mozilla%3A%3ADecoderDoctorDocumentWatcher%3A%3AAddDiagnostics&date=%3E%3D2018-01-01#graphs
*beta: https://crash-stats.mozilla.com/signature/?product=Firefox&release_channel=beta&signature=OOM%20%7C%20large%20%7C%20mozalloc_abort%20%7C%20mozalloc_handle_oom%20%7C%20moz_xrealloc%20%7C%20nsTArray_base%3CT%3E%3A%3AEnsureCapacity%3CT%3E%20%7C%20nsTArray_Impl%3CT%3E%3A%3AAppendElement%3CT%3E%20%7C%20mozilla%3A%3ADecoderDoctorDocumentWatcher%3A%3AAddDiagnostics&date=%3E%3D2018-01-01#graphs

Comment 1

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Some thoughts:

One DecoderDoctorDocumentWatcher is attached to an nsDocument on the first "interesting" event, (e.g.: canPlayType/IsTypeSupported calls, decoder issues), and will stay alive and continue gathering information until nothing more happens for at least 1 second.

So assuming this is not just unlucky (e.g., we tried to allocate a bit of memory when the system was already under stress), it means events keep happening faster that once per second, and so we never release the recorded information.

Possible solutions:
- Discard information that has been kept for more than X seconds. I don't think old information is useful in our analyses, it was intended in cases where some website would attempt to play something for a few seconds and then succeeds or gives up.
- Discard oldest reports when we've already collected lots (100+?).
- Make allocations fallible, and handle failures by just discarding everything!
The last two may indicate a high rate of events, in which case the priority should be to free resources so Firefox stays alive, instead of trying to analyze (non-critical) issues about playback.

Happy to help with questions&reviews if someone wants to tackle this. And I'll keep it in mind, if I ever find some free time.

Comment 2

[[:philipp]]

a large subset of [@ OOM | large | NS_ABORT_OOM | nsTSubstring<T>::Replace] goes through mozilla::DecoderDoctorDocumentWatcher::SynthesizeAnalysis and also spiked up in volume around june 6 (mainly on the beta channel & windows 7) , so i suppose it can be related to this bug.

https://crash-stats.mozilla.com/signature/?proto_signature=~mozilla%3A%3ADecoderDoctorDocumentWatcher%3A%3ASynthesizeAnalysis&signature=OOM%20%7C%20large%20%7C%20NS_ABORT_OOM%20%7C%20nsTSubstring%3CT%3E%3A%3AReplace&date=%3E%3D2018-06-01#reports

https://crash-stats.mozilla.com/signature/?proto_signature=~mozilla%3A%3ADecoderDoctorDocumentWatcher%3A%3ASynthesizeAnalysis&signature=OOM%20%7C%20large%20%7C%20NS_ABORT_OOM%20%7C%20nsTSubstring%3CT%3E%3A%3AReplace&date=%3E%3D2018-05-01#graphs

Comment 4

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Fixed in bug 1571648, landed in 70.