1586903 - Intermittent SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/Assertions.h:332:3 in MOZ_Crash (crashes under MediaRecorder::Session::DoSessionEndTask)

Status: RESOLVED FIXED

(Core :: Audio/Video: Recording, defect, P2)

Description

[Chris H-C :chutten]

In investigating bug 155602 I noticed there were actually two types of crash being attributed there. The reftest one is a WebRender one that I've filed already as bug 1586898. But the webplatform test one is happening with a stack containing
MediaRecorder::Session::DoSessionEndTask.

Here's the list of intermittents: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-09-30&endday=2019-10-07&tree=trunk&bug=1556021

And here's one of the stacks:

[task 2019-10-04T10:23:22.893Z] 10:23:22    ERROR - PID 15221 | ==15612==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f1012f29b97 bp 0x7ffc9c085510 sp 0x7ffc9c085510 T0)
[task 2019-10-04T10:23:22.893Z] 10:23:22     INFO - PID 15221 | ==15612==The signal is caused by a WRITE memory access.
[task 2019-10-04T10:23:22.893Z] 10:23:22     INFO - PID 15221 | ==15612==Hint: address points to the zero page.
[task 2019-10-04T10:23:23.689Z] 10:23:23     INFO - PID 15221 |     #0 0x7f1012f29b96 in MOZ_Crash /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/Assertions.h:332:3
[task 2019-10-04T10:23:23.691Z] 10:23:23     INFO - PID 15221 |     #1 0x7f1012f29b96 in InvalidArrayIndex_CRASH(unsigned long, unsigned long) /builds/worker/workspace/build/src/xpcom/ds/nsTArray.cpp:27
[task 2019-10-04T10:23:23.719Z] 10:23:23     INFO - PID 15221 |     #2 0x7f101a07aaad in ElementAt /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1065:7
[task 2019-10-04T10:23:23.722Z] 10:23:23     INFO - PID 15221 |     #3 0x7f101a07aaad in LastElement /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1109
[task 2019-10-04T10:23:23.722Z] 10:23:23     INFO - PID 15221 |     #4 0x7f101a07aaad in mozilla::dom::MediaRecorder::Session::DoSessionEndTask(nsresult)::'lambda'(mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue const&)::operator()(mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue const&) const /builds/worker/workspace/build/src/dom/media/MediaRecorder.cpp:942
[task 2019-10-04T10:23:23.722Z] 10:23:23     INFO - PID 15221 |     #5 0x7f101a079ccc in InvokeMethod<(lambda at /builds/worker/workspace/build/src/dom/media/MediaRecorder.cpp:940:16), RefPtr<mozilla::MozPromise<bool, bool, false> > ((lambda at /builds/worker/workspace/build/src/dom/media/MediaRecorder.cpp:940:16)::*)(const mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue &) const, const mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue &> /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:512:12
[task 2019-10-04T10:23:23.722Z] 10:23:23     INFO - PID 15221 |     #6 0x7f101a079ccc in InvokeCallbackMethod<true, (lambda at /builds/worker/workspace/build/src/dom/media/MediaRecorder.cpp:940:16), RefPtr<mozilla::MozPromise<bool, bool, false> > ((lambda at /builds/worker/workspace/build/src/dom/media/MediaRecorder.cpp:940:16)::*)(const mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue &) const, const mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue &, RefPtr<mozilla::MozPromise<bool, bool, false>::Private> > /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:529
[task 2019-10-04T10:23:23.722Z] 10:23:23     INFO - PID 15221 |     #7 0x7f101a079ccc in mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ThenValue<mozilla::dom::MediaRecorder::Session::DoSessionEndTask(nsresult)::'lambda'(mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue const&)>::DoResolveOrRejectInternal(mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ResolveOrRejectValue&) /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:793
[task 2019-10-04T10:23:23.722Z] 10:23:23     INFO - PID 15221 |     #8 0x7f101a072fa2 in mozilla::MozPromise<nsMainThreadPtrHandle<mozilla::dom::Blob>, nsresult, false>::ThenValueBase::ResolveOrRejectRunnable::Run() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:402:21
[task 2019-10-04T10:23:23.738Z] 10:23:23     INFO - PID 15221 |     #9 0x7f1013025141 in mozilla::SchedulerGroup::Runnable::Run() /builds/worker/workspace/build/src/xpcom/threads/SchedulerGroup.cpp:295:32
[task 2019-10-04T10:23:23.754Z] 10:23:23     INFO - PID 15221 |     #10 0x7f10130566d9 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1225:14
[task 2019-10-04T10:23:23.754Z] 10:23:23     INFO - PID 15221 |     #11 0x7f10130819a1 in NS_InvokeByIndex /builds/worker/workspace/build/src/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:106
[task 2019-10-04T10:23:23.770Z] 10:23:23     INFO - PID 15221 |     #12 0x7f1014d5bd2e in Invoke /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNative.cpp:1642:10
[task 2019-10-04T10:23:23.770Z] 10:23:23     INFO - PID 15221 |     #13 0x7f1014d5bd2e in Call /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNative.cpp:1183
[task 2019-10-04T10:23:23.771Z] 10:23:23     INFO - PID 15221 |     #14 0x7f1014d5bd2e in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNative.cpp:1149
[task 2019-10-04T10:23:23.771Z] 10:23:23     INFO - PID 15221 |     #15 0x7f1014d61eb4 in XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:946:10
[task 2019-10-04T10:23:23.787Z] 10:23:23     INFO - PID 15221 |     #16 0x7f101f9267dc in CallJSNative /builds/worker/workspace/build/src/js/src/vm/Interpreter.cpp:458:13
[task 2019-10-04T10:23:23.787Z] 10:23:23     INFO - PID 15221 |     #17 0x7f101f9267dc in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /builds/worker/workspace/build/src/js/src/vm/Interpreter.cpp:551
[task 2019-10-04T10:23:23.869Z] 10:23:23     INFO - PID 15221 |     #18 0x7f1020c16a2c in js::jit::DoCallFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICCall_Fallback*, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) /builds/worker/workspace/build/src/js/src/jit/BaselineIC.cpp:3233:10
[task 2019-10-04T10:23:23.877Z] 10:23:23     INFO - PID 15221 |     #19 0x36c1f8864797  (<unknown module>)
[task 2019-10-04T10:23:23.877Z] 10:23:23     INFO - PID 15221 | AddressSanitizer can not provide additional info.
[task 2019-10-04T10:23:23.878Z] 10:23:23     INFO - PID 15221 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/Assertions.h:332:3 in MOZ_Crash

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 2

[Intermittent Failures Robot]

1 failures in 3801 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1586903&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1586903 - Wait for track listener removals rather than just informing them of shutdown. r?bryce

This does a bit of a cleanup, where changing the notification to waiting for
removal is the major task. It also removes the special handling of not informing
listeners of shutdown on Cancel(), and a bit of cleanup around MozPromise usage.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/509719f98963
Wait for track listener removals rather than just informing them of shutdown. r=bryce

Comment 6

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/509719f98963

Comment 7

[Karl Tomlinson (:karlt)]

Looks like this crash was in the Then()-handler after DoSessionEndTask() presumably at https://hg.mozilla.org/mozilla-central/annotate/9ea7aabf10fbf326e43daaa88b30f477b4ee3c32/dom/media/MediaRecorder.cpp#l942

LastElement() attempts to use an index out of range either because there are no elements or because mSessions has been modified on another thread. I see mSessions modified only on the main thread, so I assume the former.

Sessions are removed only after Session::Shutdown(), which is typically called after DoSessionEndTask(), but Session::Shutdown() is also called from ShutdownBlocker. So the question is what prevents the Then()-handler from running after the ShutdownBlocker?

EncoderListener::Forget() may have been intended to help here, but doesn't look helpful given the pending runnables in flight after EncoderListener::Shutdown() checks whether Forget() has been called.
mShutdownBlocker is not removed until after the Shutdown() triggered from DoSessionEndTask().

The ProcessNextEvent() in the stack may be evidence that this was during async process shutdown handling.

The patch here addresses bug 1589012, but I wonder whether it did anything to address this issue.

This is a safe crash because index -1 on sEmptyTArrayHeader is a null pointer.