1808025 - OOM Crash in [@ webrtc::AlignedMalloc]

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Byron Campen [:bwc]]

Crash report: https://crash-stats.mozilla.org/report/index/d36450b0-2c6e-4da6-a6e1-346530221224

Reason: EXCEPTION_BREAKPOINT

Top 10 frames of crashing thread:

0  KERNELBASE.dll  wil::details::DebugBreak  
1  xul.dll    third_party/libwebrtc/rtc_base/checks.cc:52
2  xul.dll  rtc::webrtc_checks_impl::FatalLog  third_party/libwebrtc/rtc_base/checks.cc:170
3  xul.dll  rtc::webrtc_checks_impl::LogStreamer<>::Call  third_party/libwebrtc/rtc_base/checks.h:247
3  xul.dll  rtc::webrtc_checks_impl::LogStreamer<rtc::webrtc_checks_impl::Val<9, const char*> >::Call const  third_party/libwebrtc/rtc_base/checks.h:298
3  xul.dll  rtc::webrtc_checks_impl::FatalLogCall<0>::operator&  third_party/libwebrtc/rtc_base/checks.h:335
3  xul.dll  webrtc::AlignedMalloc  third_party/libwebrtc/rtc_base/memory/aligned_malloc.cc:83
4  xul.dll  webrtc::I420Buffer::I420Buffer  third_party/libwebrtc/api/video/i420_buffer.cc:51
4  xul.dll  rtc::RefCountedObject<webrtc::I420Buffer>::RefCountedObject  third_party/libwebrtc/rtc_base/ref_counted_object.h:34
4  xul.dll  rtc::make_ref_counted<webrtc::I420Buffer, int&, int&, int&, int&, int&, nullptr>  third_party/libwebrtc/api/make_ref_counted.h:86

This is a very common crash in the wild right now. Line number (aligned_malloc.cc:83) appears to be wrong, it is probably actually here. Maybe this is an OOM (the system memory usage is consistently in the high 80s or above), or maybe |size| is just too large.

Comment 1

[Byron Campen [:bwc]]

The fact that this crash seems to start in early December is because of bug 1800132 making libwebrtc crashes searchable that far down the stack; it is likely that this was going on well before then, although it is hard to tell exactly when.

Comment 2

[Andreas Pehrson [:pehrsons]]

I agree this must be an OOM. We (and upstream) in the capture paths tend to allocate for each buffer needed. At a minimum we should use buffer pools. The crash in comment 0 crashes here but I'm sure the site at which this OOMs is spread wide.

For instance, in the last week on crash-stats I see 147 crashes with the reported signature.

124 of these (84%) are in VideoCaptureImpl::IncomingFrame
10 of these (7%) are in LibvpxVp8Encoder::PrepareBuffers
6 of these (4%) are in DesktopCaptureImpl::IncomingFrame
4 of these (3%) are in LibvpxVp8Decoder::ReturnFrame
3 of these (2%) are in DesktopCaptureImpl::OnCaptureResult (the link also includes those 6 in IncomingFrame)

The one in LibvpxVp8Decoder::ReturnFrame is through a buffer pool, whereas the rest are not. If we fix the capture paths, that will take care of the bulk of these. The one in the camera capture path probably weighs so heavily because of 1) higher usage than desktop capture, and 2) it's first in the pipe.

We should also consider overriding libwebrtc's WriteFatalLog so we can annotate crash reports with their crash reason.

Comment 3

[Andreas Pehrson [:pehrsons]]

S2 is perhaps a bit excessive, you can't completely guard against OOM. On the other hand, most of these (the capture ones) bring down the parent process, which is unfortunate, and probably justifies S2.