1348381 - VP8TrackEncoder crash: [@mozilla::VP8TrackEncoder::PrepareRawFrame]

Status: RESOLVED FIXED

(Core :: Audio/Video: Recording, defect, P1)

Description

[Christoph Diehl [:posidron]]

Attachment: CreateBlackImage.html

[57617] WARNING: SharedPlanarYCbCrImage::Allocate failed.: file /srv/mozilla/mozilla-inbound/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp, line 196
Assertion failure: false, at /srv/mozilla/mozilla-inbound/dom/media/VideoSegment.cpp:85
#01: mozilla::VP8TrackEncoder::PrepareRawFrame(mozilla::VideoChunk&)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67c8e3c]
#02: mozilla::VP8TrackEncoder::GetEncodedTrack(mozilla::EncodedFrameContainer&)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67ce6a6]
#03: mozilla::MediaEncoder::WriteEncodedDataToMuxer(mozilla::TrackEncoder*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67b9eeb]
#04: mozilla::MediaEncoder::GetEncodedData(nsTArray<nsTArray<unsigned char> >*, nsAString_internal&)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67b7e04]
#05: mozilla::dom::MediaRecorder::Session::Extract(bool)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x66bf4f2]
#06: mozilla::dom::MediaRecorder::Session::ExtractRunnable::Run()[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x66bedc8]
#07: nsThread::ProcessNextEvent(bool, bool*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x3133ab]
#08: NS_ProcessNextEvent(nsIThread*, bool)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x30e8d7]
#09: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x134b49c]
#10: MessageLoop::RunInternal()[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x1260c18]
#11: MessageLoop::Run()[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x12608dd]
#12: nsThread::ThreadFunc(void*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x309aea]
#13: _pt_root[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib +0x3694aa]
#14: _pthread_body[/usr/lib/system/libsystem_pthread.dylib +0x3aab]
#15: _pthread_body[/usr/lib/system/libsystem_pthread.dylib +0x39f7]
ASAN:DEADLYSIGNAL
=================================================================
==57617==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00011f472e62 bp 0x7000071858d0 sp 0x700007185740 T99)
#0 0x11f472e61 in mozilla::VideoFrame::CreateBlackImage(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x675fe61)
#1 0x11f4dbe3b in mozilla::VP8TrackEncoder::PrepareRawFrame(mozilla::VideoChunk&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67c8e3b)
#2 0x11f4e16a5 in mozilla::VP8TrackEncoder::GetEncodedTrack(mozilla::EncodedFrameContainer&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67ce6a5)
#3 0x11f4cceea in mozilla::MediaEncoder::WriteEncodedDataToMuxer(mozilla::TrackEncoder*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67b9eea)
#4 0x11f4cae03 in mozilla::MediaEncoder::GetEncodedData(nsTArray<nsTArray<unsigned char> >*, nsAString_internal&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67b7e03)
#5 0x11f3d24f1 in mozilla::dom::MediaRecorder::Session::Extract(bool) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66bf4f1)
#6 0x11f3d1dc7 in mozilla::dom::MediaRecorder::Session::ExtractRunnable::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66bedc7)
#7 0x1190263aa in nsThread::ProcessNextEvent(bool, bool*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x3133aa)
#8 0x1190218d6 in NS_ProcessNextEvent(nsIThread*, bool) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x30e8d6)
#9 0x11a05e49b in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x134b49b)
#10 0x119f73c17 in MessageLoop::RunInternal() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x1260c17)
#11 0x119f738dc in MessageLoop::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x12608dc)
#12 0x11901cae9 in nsThread::ThreadFunc(void*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x309ae9)
#13 0x10ce3b4a9 in _pt_root (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib+0x3694a9)
#14 0x7fffaae69aaa in _pthread_body (/usr/lib/system/libsystem_pthread.dylib+0x3aaa)
#15 0x7fffaae699f6 in _pthread_start (/usr/lib/system/libsystem_pthread.dylib+0x39f6)
#16 0x7fffaae691fc in thread_start (/usr/lib/system/libsystem_pthread.dylib+0x31fc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x675fe61) in mozilla::VideoFrame::CreateBlackImage(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&)
Thread T99 created by T0 here:
#0 0x10976f379 in wrap_pthread_create (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib+0x41379)
#1 0x10ce37c1a in _PR_CreateThread (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib+0x365c1a)
#2 0x10ce377da in PR_CreateThread (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib+0x3657da)
#3 0x11901f72a in nsThread::Init(nsACString_internal const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x30c72a)
#4 0x11902c346 in nsThreadManager::NewNamedThread(nsACString_internal const&, unsigned int, nsIThread**) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x319346)
#5 0x11902f2f6 in NS_NewNamedThread(nsACString_internal const&, nsIThread**, nsIRunnable*, unsigned int) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x31c2f6)
#6 0x11f3cb51e in mozilla::dom::MediaRecorder::Session::InitEncoder(unsigned char, int) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66b851e)
#7 0x11f3cca47 in mozilla::dom::MediaRecorder::Session::TracksAvailableCallback::NotifyTracksAvailable(mozilla::DOMMediaStream*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66b9a47)
#8 0x11f0ecc26 in mozilla::DOMMediaStream::CheckTracksAvailable() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x63d9c26)
#9 0x11f10d567 in mozilla::detail::RunnableMethodImpl<mozilla::DOMMediaStream*, void (mozilla::DOMMediaStream::*)(), true, false>::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x63fa567)
#10 0x1190263aa in nsThread::ProcessNextEvent(bool, bool*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x3133aa)
#11 0x11901e580 in NS_ProcessPendingEvents(nsIThread*, unsigned int) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x30b580)
#12 0x120ab23cf in nsBaseAppShell::NativeEventCallback() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7d9f3cf)
#13 0x120bc3555 in nsAppShell::ProcessGeckoEvents(void*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7eb0555)
#14 0x7fff9571a980 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0xa7980)
#15 0x7fff956fb9f6 in __CFRunLoopDoSources0 (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0x889f6)
#16 0x7fff956faf75 in __CFRunLoopRun (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0x87f75)
#17 0x7fff956fa973 in CFRunLoopRunSpecific (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0x87973)
#18 0x7fff94c86a5b in RunCurrentEventLoopInMode (/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox+0x30a5b)
#19 0x7fff94c86890 in ReceiveNextEventCommon (/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox+0x30890)
#20 0x7fff94c866c5 in _BlockUntilNextEventMatchingListInModeWithFilter (/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox+0x306c5)
#21 0x7fff9322c5b3 in _DPSNextEvent (/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit+0x475b3)
#22 0x7fff939a6d6a in -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] (/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit+0x7c1d6a)
#23 0x120bc1a6c in -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7eaea6c)
#24 0x7fff93220f34 in -[NSApplication run] (/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit+0x3bf34)
#25 0x120bc45e6 in nsAppShell::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7eb15e6)
#26 0x124800c8d in nsAppStartup::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbaedc8d)
#27 0x124a20795 in XREMain::XRE_mainRun() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbd0d795)
#28 0x124a2376a in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbd1076a)
#29 0x124a24ddd in XRE_main(int, char**, mozilla::BootstrapConfig const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbd11ddd)
#30 0x1016170df in main (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/firefox+0x1000020df)
#31 0x101616993 in start (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/firefox+0x100001993)


Command: /srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/firefox -width 512 -height 512 /srv/logs/framboise/framboise_Fri_Mar_17_13-45-14_2017/faults/0/testcase.html

Comment 1

[Alex Chronopoulos [:achronop]]

Andreas, is it something in your area?

Comment 2

[Andreas Pehrson [:pehrsons]]

It is. Making the encoder skip the frame if allocation fails seems like a better solution here. But perhaps we do in non-debug builds? I imagine this could happen with peerconnections and other consumers as well.

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

Review commit: https://reviewboard.mozilla.org/r/127158/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127158/

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1348381 - Add crashtest.

Review commit: https://reviewboard.mozilla.org/r/134722/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/134722/

Comment 5

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127158/diff/1-2/

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

https://reviewboard.mozilla.org/r/127158/#review137696

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 8862807 [details]
Bug 1348381 - Add crashtest.

https://reviewboard.mozilla.org/r/134722/#review137698

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/47debe9bbc71
Add crashtest. r=jesup
https://hg.mozilla.org/integration/autoland/rev/e1a305a0f637
Handle VideoFrame::CreateBlackImage returning nullptr. r=jesup

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/47debe9bbc71
https://hg.mozilla.org/mozilla-central/rev/e1a305a0f637

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Mozregression says this was a regression from bug 1201363, FWIW. Please request Beta and ESR52 approval on this when you get a chance.

Comment 11

[Andreas Pehrson [:pehrsons]]

Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: An application could segfault us at will.
User impact if declined: An application could segfault us at will.
Fix Landed on Version: 55
Risk to taking this patch (and alternatives if risky): Low. Patch adds a simple early-exit path. Test case makes sure it works.
String or UUID changes made by this patch: None.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1201363
[User impact if declined]: An application could segfault us at will.
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Patch adds a simple early-exit path. Test case makes sure it works.
[String changes made/needed]: None

Comment 12

[Gerry Chang [:gchang]]

Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

Fix a crash. Beta54+. Should be in 54 beta 6.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/3f63049dfca9
https://hg.mozilla.org/releases/mozilla-beta/rev/b13c3ced8ccf

Comment 14

[Andrei Vaida [:avaida]]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #11)
> [Is this code covered by automated tests?]: Yes
> [Has the fix been verified in Nightly?]: Yes
> [Needs manual test from QE? If yes, steps to reproduce]: No

Setting qe-verify- based on Andreas' assessment on manual testing needs and the fact that this fix has automated coverage.

Comment 15

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

It's not clear whether this is a top crasher, doesn't seem like it. For ESR, we typically accept sec-high, sec-crits, severe recent regressions, severe stability issues and as such this doesn't meet that bar.