VP8TrackEncoder crash: [@mozilla::VP8TrackEncoder::PrepareRawFrame]

RESOLVED FIXED in Firefox 54

Status

()

Core
Audio/Video: Recording
P1
critical
Rank:
15
RESOLVED FIXED
10 months ago
8 months ago

People

(Reporter: posidron, Assigned: pehrsons)

Tracking

({crash, testcase})

51 Branch
mozilla55
crash, testcase
Points:
---
Bug Flags:
in-testsuite +
qe-verify -

Firefox Tracking Flags

(firefox-esr45 unaffected, firefox-esr52 wontfix, firefox53 wontfix, firefox54 fixed, firefox55 fixed)

Details

MozReview Requests

()

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

(Reporter)

Description

10 months ago
Created attachment 8848604 [details]
CreateBlackImage.html

[57617] WARNING: SharedPlanarYCbCrImage::Allocate failed.: file /srv/mozilla/mozilla-inbound/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp, line 196
Assertion failure: false, at /srv/mozilla/mozilla-inbound/dom/media/VideoSegment.cpp:85
#01: mozilla::VP8TrackEncoder::PrepareRawFrame(mozilla::VideoChunk&)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67c8e3c]
#02: mozilla::VP8TrackEncoder::GetEncodedTrack(mozilla::EncodedFrameContainer&)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67ce6a6]
#03: mozilla::MediaEncoder::WriteEncodedDataToMuxer(mozilla::TrackEncoder*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67b9eeb]
#04: mozilla::MediaEncoder::GetEncodedData(nsTArray<nsTArray<unsigned char> >*, nsAString_internal&)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x67b7e04]
#05: mozilla::dom::MediaRecorder::Session::Extract(bool)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x66bf4f2]
#06: mozilla::dom::MediaRecorder::Session::ExtractRunnable::Run()[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x66bedc8]
#07: nsThread::ProcessNextEvent(bool, bool*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x3133ab]
#08: NS_ProcessNextEvent(nsIThread*, bool)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x30e8d7]
#09: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x134b49c]
#10: MessageLoop::RunInternal()[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x1260c18]
#11: MessageLoop::Run()[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x12608dd]
#12: nsThread::ThreadFunc(void*)[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL +0x309aea]
#13: _pt_root[/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib +0x3694aa]
#14: _pthread_body[/usr/lib/system/libsystem_pthread.dylib +0x3aab]
#15: _pthread_body[/usr/lib/system/libsystem_pthread.dylib +0x39f7]
ASAN:DEADLYSIGNAL
=================================================================
==57617==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00011f472e62 bp 0x7000071858d0 sp 0x700007185740 T99)
#0 0x11f472e61 in mozilla::VideoFrame::CreateBlackImage(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x675fe61)
#1 0x11f4dbe3b in mozilla::VP8TrackEncoder::PrepareRawFrame(mozilla::VideoChunk&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67c8e3b)
#2 0x11f4e16a5 in mozilla::VP8TrackEncoder::GetEncodedTrack(mozilla::EncodedFrameContainer&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67ce6a5)
#3 0x11f4cceea in mozilla::MediaEncoder::WriteEncodedDataToMuxer(mozilla::TrackEncoder*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67b9eea)
#4 0x11f4cae03 in mozilla::MediaEncoder::GetEncodedData(nsTArray<nsTArray<unsigned char> >*, nsAString_internal&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x67b7e03)
#5 0x11f3d24f1 in mozilla::dom::MediaRecorder::Session::Extract(bool) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66bf4f1)
#6 0x11f3d1dc7 in mozilla::dom::MediaRecorder::Session::ExtractRunnable::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66bedc7)
#7 0x1190263aa in nsThread::ProcessNextEvent(bool, bool*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x3133aa)
#8 0x1190218d6 in NS_ProcessNextEvent(nsIThread*, bool) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x30e8d6)
#9 0x11a05e49b in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x134b49b)
#10 0x119f73c17 in MessageLoop::RunInternal() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x1260c17)
#11 0x119f738dc in MessageLoop::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x12608dc)
#12 0x11901cae9 in nsThread::ThreadFunc(void*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x309ae9)
#13 0x10ce3b4a9 in _pt_root (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib+0x3694a9)
#14 0x7fffaae69aaa in _pthread_body (/usr/lib/system/libsystem_pthread.dylib+0x3aaa)
#15 0x7fffaae699f6 in _pthread_start (/usr/lib/system/libsystem_pthread.dylib+0x39f6)
#16 0x7fffaae691fc in thread_start (/usr/lib/system/libsystem_pthread.dylib+0x31fc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x675fe61) in mozilla::VideoFrame::CreateBlackImage(mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> const&)
Thread T99 created by T0 here:
#0 0x10976f379 in wrap_pthread_create (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib+0x41379)
#1 0x10ce37c1a in _PR_CreateThread (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib+0x365c1a)
#2 0x10ce377da in PR_CreateThread (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib+0x3657da)
#3 0x11901f72a in nsThread::Init(nsACString_internal const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x30c72a)
#4 0x11902c346 in nsThreadManager::NewNamedThread(nsACString_internal const&, unsigned int, nsIThread**) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x319346)
#5 0x11902f2f6 in NS_NewNamedThread(nsACString_internal const&, nsIThread**, nsIRunnable*, unsigned int) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x31c2f6)
#6 0x11f3cb51e in mozilla::dom::MediaRecorder::Session::InitEncoder(unsigned char, int) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66b851e)
#7 0x11f3cca47 in mozilla::dom::MediaRecorder::Session::TracksAvailableCallback::NotifyTracksAvailable(mozilla::DOMMediaStream*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x66b9a47)
#8 0x11f0ecc26 in mozilla::DOMMediaStream::CheckTracksAvailable() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x63d9c26)
#9 0x11f10d567 in mozilla::detail::RunnableMethodImpl<mozilla::DOMMediaStream*, void (mozilla::DOMMediaStream::*)(), true, false>::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x63fa567)
#10 0x1190263aa in nsThread::ProcessNextEvent(bool, bool*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x3133aa)
#11 0x11901e580 in NS_ProcessPendingEvents(nsIThread*, unsigned int) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x30b580)
#12 0x120ab23cf in nsBaseAppShell::NativeEventCallback() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7d9f3cf)
#13 0x120bc3555 in nsAppShell::ProcessGeckoEvents(void*) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7eb0555)
#14 0x7fff9571a980 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0xa7980)
#15 0x7fff956fb9f6 in __CFRunLoopDoSources0 (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0x889f6)
#16 0x7fff956faf75 in __CFRunLoopRun (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0x87f75)
#17 0x7fff956fa973 in CFRunLoopRunSpecific (/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation+0x87973)
#18 0x7fff94c86a5b in RunCurrentEventLoopInMode (/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox+0x30a5b)
#19 0x7fff94c86890 in ReceiveNextEventCommon (/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox+0x30890)
#20 0x7fff94c866c5 in _BlockUntilNextEventMatchingListInModeWithFilter (/System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox+0x306c5)
#21 0x7fff9322c5b3 in _DPSNextEvent (/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit+0x475b3)
#22 0x7fff939a6d6a in -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] (/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit+0x7c1d6a)
#23 0x120bc1a6c in -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7eaea6c)
#24 0x7fff93220f34 in -[NSApplication run] (/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit+0x3bf34)
#25 0x120bc45e6 in nsAppShell::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0x7eb15e6)
#26 0x124800c8d in nsAppStartup::Run() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbaedc8d)
#27 0x124a20795 in XREMain::XRE_mainRun() (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbd0d795)
#28 0x124a2376a in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbd1076a)
#29 0x124a24ddd in XRE_main(int, char**, mozilla::BootstrapConfig const&) (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/XUL+0xbd11ddd)
#30 0x1016170df in main (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/firefox+0x1000020df)
#31 0x101616993 in start (/srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/firefox+0x100001993)


Command: /srv/mozilla/mozilla-inbound/ff-asan-debug/dist/NightlyDebug.app/Contents/MacOS/firefox -width 512 -height 512 /srv/logs/framboise/framboise_Fri_Mar_17_13-45-14_2017/faults/0/testcase.html
Andreas, is it something in your area?
Rank: 15
Flags: needinfo?(pehrson)
Priority: -- → P1
(Assignee)

Comment 2

10 months ago
It is. Making the encoder skip the frame if allocation fails seems like a better solution here. But perhaps we do in non-debug builds? I imagine this could happen with peerconnections and other consumers as well.
Assignee: nobody → pehrson
Flags: needinfo?(pehrson)

Updated

9 months ago
See Also: → bug 1356113
Comment hidden (mozreview-request)
(Assignee)

Updated

9 months ago
Status: NEW → ASSIGNED
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 6

9 months ago
mozreview-review
Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

https://reviewboard.mozilla.org/r/127158/#review137696
Attachment #8862795 - Flags: review?(rjesup) → review+

Comment 7

9 months ago
mozreview-review
Comment on attachment 8862807 [details]
Bug 1348381 - Add crashtest.

https://reviewboard.mozilla.org/r/134722/#review137698
Attachment #8862807 - Flags: review?(rjesup) → review+

Comment 8

9 months ago
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/47debe9bbc71
Add crashtest. r=jesup
https://hg.mozilla.org/integration/autoland/rev/e1a305a0f637
Handle VideoFrame::CreateBlackImage returning nullptr. r=jesup

Comment 9

9 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/47debe9bbc71
https://hg.mozilla.org/mozilla-central/rev/e1a305a0f637
Status: ASSIGNED → RESOLVED
Last Resolved: 9 months ago
status-firefox55: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Mozregression says this was a regression from bug 1201363, FWIW. Please request Beta and ESR52 approval on this when you get a chance.
Blocks: 1201363
status-firefox53: --- → wontfix
status-firefox54: --- → affected
status-firefox-esr45: --- → unaffected
status-firefox-esr52: --- → affected
Flags: in-testsuite+
Version: Trunk → 51 Branch
(Assignee)

Comment 11

9 months ago
Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration: An application could segfault us at will.
User impact if declined: An application could segfault us at will.
Fix Landed on Version: 55
Risk to taking this patch (and alternatives if risky): Low. Patch adds a simple early-exit path. Test case makes sure it works.
String or UUID changes made by this patch: None.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1201363
[User impact if declined]: An application could segfault us at will.
[Is this code covered by automated tests?]: Yes
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: No
[Why is the change risky/not risky?]: Patch adds a simple early-exit path. Test case makes sure it works.
[String changes made/needed]: None
Attachment #8862795 - Flags: approval-mozilla-esr52?
Attachment #8862795 - Flags: approval-mozilla-beta?
Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

Fix a crash. Beta54+. Should be in 54 beta 6.
Attachment #8862795 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #11)
> [Is this code covered by automated tests?]: Yes
> [Has the fix been verified in Nightly?]: Yes
> [Needs manual test from QE? If yes, steps to reproduce]: No

Setting qe-verify- based on Andreas' assessment on manual testing needs and the fact that this fix has automated coverage.
Flags: qe-verify-

Comment 15

8 months ago
Comment on attachment 8862795 [details]
Bug 1348381 - Handle VideoFrame::CreateBlackImage returning nullptr.

It's not clear whether this is a top crasher, doesn't seem like it. For ESR, we typically accept sec-high, sec-crits, severe recent regressions, severe stability issues and as such this doesn't meet that bar.
Attachment #8862795 - Flags: approval-mozilla-esr52? → approval-mozilla-esr52-

Updated

8 months ago
status-firefox-esr52: affected → wontfix
You need to log in before you can comment on or make changes to this bug.