Permafailing tier 2 Assertion failure: Request::mDisconnected, at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:429

RESOLVED FIXED in Firefox 66

Status

()

defect
P1
normal
Rank:
10
RESOLVED FIXED
6 months ago
5 months ago

People

(Reporter: intermittent-bug-filer, Assigned: jib)

Tracking

({intermittent-failure, regression})

unspecified
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox65 wontfix, firefox66 fixed, firefox67 fixed)

Details

(Whiteboard: [stockwell fixed])

Attachments

(1 attachment, 1 obsolete attachment)

Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=223972870&repo=mozilla-central

https://queue.taskcluster.net/v1/task/H4vQcnknRjeIX5EBIb5nPQ/runs/0/artifacts/public/logs/live_backing.log

Created based on https://bugzilla.mozilla.org/show_bug.cgi?id=1346506#c9

[task 2019-01-25T07:48:53.624Z] 07:48:53 INFO - TEST-START | /mediacapture-streams/MediaStreamTrack-MediaElement-disabled-audio-is-silence.https.html
[task 2019-01-25T07:48:53.645Z] 07:48:53 INFO - PID 5440 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-01-25T07:48:54.631Z] 07:48:54 INFO - PID 5440 | -----------------------------------------------------
[task 2019-01-25T07:48:54.632Z] 07:48:54 INFO - PID 5440 | Suppressions used:
[task 2019-01-25T07:48:54.632Z] 07:48:54 INFO - PID 5440 | count bytes template
[task 2019-01-25T07:48:54.634Z] 07:48:54 INFO - PID 5440 | 500 15968 nsComponentManagerImpl
[task 2019-01-25T07:48:54.634Z] 07:48:54 INFO - PID 5440 | 2 416 mozJSComponentLoader::LoadModule
[task 2019-01-25T07:48:54.634Z] 07:48:54 INFO - PID 5440 | 611 17713 libfontconfig.so
[task 2019-01-25T07:48:54.634Z] 07:48:54 INFO - PID 5440 | 1 29 libglib-2.0.so
[task 2019-01-25T07:48:54.634Z] 07:48:54 INFO - PID 5440 | -----------------------------------------------------
[task 2019-01-25T07:48:54.742Z] 07:48:54 INFO - PID 5440 | ###!!! [Child][MessageChannel] Error: (msgtype=0x350100,name=PContent::Msg_AccumulateChildHistograms) Closed channel: cannot send/recv
[task 2019-01-25T07:48:54.743Z] 07:48:54 INFO - PID 5440 | ###!!! [Child][MessageChannel] Error: (msgtype=0x350101,name=PContent::Msg_AccumulateChildKeyedHistograms) Closed channel: cannot send/recv
[task 2019-01-25T07:48:54.743Z] 07:48:54 INFO - PID 5440 | ###!!! [Child][MessageChannel] Error: (msgtype=0x350105,name=PContent::Msg_RecordDiscardedData) Closed channel: cannot send/recv
[task 2019-01-25T07:48:54.880Z] 07:48:54 INFO - PID 5440 | Assertion failure: Request::mDisconnected, at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:429
[task 2019-01-25T07:48:54.881Z] 07:48:54 INFO - PID 5440 | AddressSanitizer:DEADLYSIGNAL
[task 2019-01-25T07:48:54.881Z] 07:48:54 INFO - PID 5440 | =================================================================
[task 2019-01-25T07:48:54.881Z] 07:48:54 ERROR - PID 5440 | ==6464==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fc386da11e8 bp 0x7ffc436ef450 sp 0x7ffc436ef440 T0)
[task 2019-01-25T07:48:54.882Z] 07:48:54 INFO - PID 5440 | ==6464==The signal is caused by a WRITE memory access.
[task 2019-01-25T07:48:54.882Z] 07:48:54 INFO - PID 5440 | ==6464==Hint: address points to the zero page.
[task 2019-01-25T07:48:55.033Z] 07:48:55 INFO - Closing window 55834574849
[task 2019-01-25T07:48:55.098Z] 07:48:55 INFO - PID 5440 | JavaScript error: , line 0: InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable
[task 2019-01-25T07:48:55.129Z] 07:48:55 INFO - .
[task 2019-01-25T07:48:55.130Z] 07:48:55 INFO - TEST-OK | /mediacapture-streams/MediaStreamTrack-MediaElement-disabled-audio-is-silence.https.html | took 1505ms
[task 2019-01-25T07:48:55.131Z] 07:48:55 INFO - TEST-START | /mediacapture-streams/MediaStreamTrack-MediaElement-disabled-video-is-black.https.html
[task 2019-01-25T07:48:55.153Z] 07:48:55 INFO - PID 5440 | SharedThreadPool in xpcom-shutdown-threads. Waiting for pools "CubebOperation"
[task 2019-01-25T07:48:55.706Z] 07:48:55 INFO - PID 5440 | #0 0x7fc386da11e7 in mozilla::MozPromise<bool, nsresult, true>::ThenValueBase::AssertIsDead() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:429:9
[task 2019-01-25T07:48:55.708Z] 07:48:55 INFO - PID 5440 | #1 0x7fc386da1870 in mozilla::MozPromise<bool, nsresult, true>::AssertIsDead() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:945:13
[task 2019-01-25T07:48:55.710Z] 07:48:55 INFO - PID 5440 | #2 0x7fc386da1b88 in mozilla::MozPromise<bool, nsresult, true>::~MozPromise() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:986:5
[task 2019-01-25T07:48:55.711Z] 07:48:55 INFO - PID 5440 | #3 0x7fc386da16fd in mozilla::MozPromise<bool, nsresult, true>::Private::~Private() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:242:9
[task 2019-01-25T07:48:55.733Z] 07:48:55 INFO - PID 5440 | #4 0x7fc38bc9fcc0 in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:137:3
[task 2019-01-25T07:48:55.733Z] 07:48:55 INFO - PID 5440 | #5 0x7fc38bc9fcc0 in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:45
[task 2019-01-25T07:48:55.733Z] 07:48:55 INFO - PID 5440 | #6 0x7fc38bc9fcc0 in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:362
[task 2019-01-25T07:48:55.733Z] 07:48:55 INFO - PID 5440 | #7 0x7fc38bc9fcc0 in ~RefPtr /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:76
[task 2019-01-25T07:48:55.733Z] 07:48:55 INFO - PID 5440 | #8 0x7fc38bc9fcc0 in ~MozPromiseHolder /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:1116
[task 2019-01-25T07:48:55.734Z] 07:48:55 INFO - PID 5440 | #9 0x7fc38bc9fcc0 in ~ /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:3042
[task 2019-01-25T07:48:55.734Z] 07:48:55 INFO - PID 5440 | #10 0x7fc38bc9fcc0 in ~LambdaRunnable /builds/worker/workspace/build/src/dom/media/systemservices/MediaUtils.h:66
[task 2019-01-25T07:48:55.734Z] 07:48:55 INFO - PID 5440 | #11 0x7fc38bc9fcc0 in mozilla::media::LambdaRunnable<mozilla::MediaInputPort::BlockSourceTrackId(int, mozilla::BlockingMode)::$_0>::~LambdaRunnable() /builds/worker/workspace/build/src/dom/media/systemservices/MediaUtils.h:66
[task 2019-01-25T07:48:55.749Z] 07:48:55 INFO - PID 5440 | #12 0x7fc3851b378c in mozilla::Runnable::Release() /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:49:1
[task 2019-01-25T07:48:55.751Z] 07:48:55 INFO - PID 5440 | #13 0x7fc38518163d in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:45:40
[task 2019-01-25T07:48:55.752Z] 07:48:55 INFO - PID 5440 | #14 0x7fc38518163d in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:362
[task 2019-01-25T07:48:55.754Z] 07:48:55 INFO - PID 5440 | #15 0x7fc38518163d in ~RefPtr /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:76
[task 2019-01-25T07:48:55.755Z] 07:48:55 INFO - PID 5440 | #16 0x7fc38518163d in ~Runner /builds/worker/workspace/build/src/xpcom/threads/AbstractThread.cpp:109
[task 2019-01-25T07:48:55.755Z] 07:48:55 INFO - PID 5440 | #17 0x7fc38518163d in mozilla::EventTargetWrapper::Runner::~Runner() /builds/worker/workspace/build/src/xpcom/threads/AbstractThread.cpp:109
[task 2019-01-25T07:48:55.757Z] 07:48:55 INFO - PID 5440 | #18 0x7fc3851b3b4c in Release /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:49:1
[task 2019-01-25T07:48:55.758Z] 07:48:55 INFO - PID 5440 | #19 0x7fc3851b3b4c in mozilla::CancelableRunnable::Release() /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:71
[task 2019-01-25T07:48:55.760Z] 07:48:55 INFO - PID 5440 | #20 0x7fc38bc9598c in ~nsCOMPtr_base /builds/worker/workspace/build/src/obj-firefox/dist/include/nsCOMPtr.h:331:7
[task 2019-01-25T07:48:55.761Z] 07:48:55 INFO - PID 5440 | #21 0x7fc38bc9598c in Destruct /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:522
[task 2019-01-25T07:48:55.761Z] 07:48:55 INFO - PID 5440 | #22 0x7fc38bc9598c in DestructRange /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:2178
[task 2019-01-25T07:48:55.762Z] 07:48:55 INFO - PID 5440 | #23 0x7fc38bc9598c in ClearAndRetainStorage /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1296
[task 2019-01-25T07:48:55.763Z] 07:48:55 INFO - PID 5440 | #24 0x7fc38bc9598c in Clear /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1759
[task 2019-01-25T07:48:55.766Z] 07:48:55 INFO - PID 5440 | #25 0x7fc38bc9598c in mozilla::(anonymous namespace)::MediaStreamGraphShutDownRunnable::Run() /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:1546
[task 2019-01-25T07:48:55.767Z] 07:48:55 INFO - PID 5440 | #26 0x7fc3851848f6 in mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/TaskDispatcher.h:197:35
[task 2019-01-25T07:48:55.768Z] 07:48:55 INFO - PID 5440 | #27 0x7fc385180e1b in mozilla::EventTargetWrapper::Runner::Run() /builds/worker/workspace/build/src/xpcom/threads/AbstractThread.cpp:139:32
[task 2019-01-25T07:48:55.769Z] 07:48:55 INFO - PID 5440 | #28 0x7fc385176901 in mozilla::SchedulerGroup::Runnable::Run() /builds/worker/workspace/build/src/xpcom/threads/SchedulerGroup.cpp:299:32
[task 2019-01-25T07:48:55.769Z] 07:48:55 INFO - PID 5440 | #29 0x7fc3851a69d6 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1160:14
[task 2019-01-25T07:48:55.770Z] 07:48:55 INFO - PID 5440 | #30 0x7fc3851d0af1 in NS_InvokeByIndex /builds/worker/workspace/build/src/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:106
[task 2019-01-25T07:48:55.780Z] 07:48:55 INFO - PID 5440 | #31 0x7fc386c98113 in Invoke /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNative.cpp:1628:10
[task 2019-01-25T07:48:55.781Z] 07:48:55 INFO - PID 5440 | #32 0x7fc386c98113 in Call /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNative.cpp:1186
[task 2019-01-25T07:48:55.781Z] 07:48:55 INFO - PID 5440 | #33 0x7fc386c98113 in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNative.cpp:1152
[task 2019-01-25T07:48:55.784Z] 07:48:55 INFO - PID 5440 | #34 0x7fc386c9e166 in XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*) /builds/worker/workspace/build/src/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:946:10
[task 2019-01-25T07:48:55.802Z] 07:48:55 INFO - PID 5440 | #35 0x194c174b2c0f (<unknown module>)
[task 2019-01-25T07:48:55.802Z] 07:48:55 INFO - PID 5440 | AddressSanitizer can not provide additional info.
[task 2019-01-25T07:48:55.802Z] 07:48:55 INFO - PID 5440 | SUMMARY: AddressSanitizer: SEGV /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:429:9 in mozilla::MozPromise<bool, nsresult, true>::ThenValueBase::AssertIsDead()
[task 2019-01-25T07:48:55.802Z] 07:48:55 INFO - PID 5440 | ==6464==ABORTING

Component: XPCOM → Audio/Video
Component: Audio/Video → Audio/Video: Recording

Here's Cosmin's comment from bug 1346506:

(In reply to Cosmin Sabou [:CosminS] from comment #7)

Hi! This started permafailing after this push https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=2072956ca0421c3eb1835522546556bb00bf8953

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=linux%2Cx64%2Casan%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-linux64-asan%2Fopt-web-platform-tests-e10s-10%2Cw-e10s%28wpt10%29&fromchange=844767deacdb7d0335824703238c941a52a306a5&selectedJob=

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=223501540&repo=mozilla-inbound&lineNumber=13957

It fails only on Linux x64 asan, sometimes it fails without the assertion but we then have AddressSanitizer: SEGV /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:429:9 in mozilla::MozPromise<bool

James please take a look over this. Thank you.

It looks like a new test was merged in from upstream, which hit this release assertion.

Flags: needinfo?(james)
Rank: 25
Priority: P5 → P3
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There are 173 total failures in the last 7 days, all on linux64 asan.

Most recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=225673023&repo=autoland&lineNumber=14027

[task 2019-02-02T21:49:42.793Z] 21:49:42 INFO - TEST-START | /mediacapture-streams/MediaStreamTrack-MediaElement-disabled-audio-is-silence.https.html
[task 2019-02-02T21:49:43.784Z] 21:49:43 INFO - PID 5159 | -----------------------------------------------------
[task 2019-02-02T21:49:43.785Z] 21:49:43 INFO - PID 5159 | Suppressions used:
[task 2019-02-02T21:49:43.786Z] 21:49:43 INFO - PID 5159 | count bytes template
[task 2019-02-02T21:49:43.786Z] 21:49:43 INFO - PID 5159 | 242 7712 nsComponentManagerImpl
[task 2019-02-02T21:49:43.786Z] 21:49:43 INFO - PID 5159 | 2 416 mozJSComponentLoader::LoadModule
[task 2019-02-02T21:49:43.786Z] 21:49:43 INFO - PID 5159 | 611 17713 libfontconfig.so
[task 2019-02-02T21:49:43.787Z] 21:49:43 INFO - PID 5159 | 1 29 libglib-2.0.so
[task 2019-02-02T21:49:43.787Z] 21:49:43 INFO - PID 5159 | -----------------------------------------------------
[task 2019-02-02T21:49:43.909Z] 21:49:43 INFO - PID 5159 | ###!!! [Child][MessageChannel] Error: (msgtype=0x350100,name=PContent::Msg_AccumulateChildHistograms) Closed channel: cannot send/recv
[task 2019-02-02T21:49:43.911Z] 21:49:43 INFO - PID 5159 | ###!!! [Child][MessageChannel] Error: (msgtype=0x350101,name=PContent::Msg_AccumulateChildKeyedHistograms) Closed channel: cannot send/recv
[task 2019-02-02T21:49:43.911Z] 21:49:43 INFO - PID 5159 | ###!!! [Child][MessageChannel] Error: (msgtype=0x350105,name=PContent::Msg_RecordDiscardedData) Closed channel: cannot send/recv
[task 2019-02-02T21:49:44.054Z] 21:49:44 INFO - PID 5159 | Assertion failure: Request::mDisconnected, at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:429
[task 2019-02-02T21:49:44.055Z] 21:49:44 INFO - PID 5159 | AddressSanitizer:DEADLYSIGNAL
[task 2019-02-02T21:49:44.056Z] 21:49:44 INFO - PID 5159 | =================================================================
[task 2019-02-02T21:49:44.057Z] 21:49:44 ERROR - PID 5159 | ==6170==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f2f9afff878 bp 0x7ffc22ca8730 sp 0x7ffc22ca8720 T0)
[task 2019-02-02T21:49:44.058Z] 21:49:44 INFO - PID 5159 | ==6170==The signal is caused by a WRITE memory access.
[task 2019-02-02T21:49:44.058Z] 21:49:44 INFO - PID 5159 | ==6170==Hint: address points to the zero page.
[task 2019-02-02T21:49:44.205Z] 21:49:44 INFO - Closing window 55834574849
[task 2019-02-02T21:49:44.262Z] 21:49:44 INFO - PID 5159 | JavaScript error: , line 0: InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable
[task 2019-02-02T21:49:44.292Z] 21:49:44 INFO - .
[task 2019-02-02T21:49:44.295Z] 21:49:44 INFO - TEST-OK | /mediacapture-streams/MediaStreamTrack-MediaElement-disabled-audio-is-silence.https.html | took 1496ms

Blocks: 1524642

This is blocking the re-enablement of a number of webrtc wpt (bug 1524642).

Rank: 25 → 10
Priority: P3 → P1
Assignee: nobody → jib

A timing-based shutdown crash. I'm not seeing any crash signatures from the field, so not tracking atm.

Component: Audio/Video: Recording → Audio/Video: MediaStreamGraph
Component: Audio/Video: MediaStreamGraph → WebRTC: Audio/Video
See Also: → 1522448
Pushed by jbruaroey@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8853735bc243
Reject any outstanding DOMMediaStream::TrackPort::BlockSourceTrackId() promise on teardown to avoid recent debug assert permafail. r=bwc
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67

Hi Jan-Ivar, is this worth a backport to Beta?

Flags: needinfo?(jib)

Comment on attachment 9041249 [details]
Bug 1522773 - Reject any outstanding DOMMediaStream::TrackPort::BlockSourceTrackId() promise on teardown to avoid recent debug assert permafail.

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1497175

User impact if declined

I doubt it will have any impact, since it's a debug correctness assert about not leaving a MozPromise pending on teardown. No particular cleanup is waiting on this particular promise that I know of, and we haven't seen any in-field reports of crashes that sound like this, but there's a chance I'm wrong, so given a choice I'd pick the fixed code we know is correct, over the code that probably cleans itself up just as well maybe.

Is this code covered by automated tests?

Yes

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

Fairly straightforward fix; common pattern used elsewhere.

String changes made/needed

Flags: needinfo?(jib)
Attachment #9041249 - Flags: approval-mozilla-beta?

Comment on attachment 9041249 [details]
Bug 1522773 - Reject any outstanding DOMMediaStream::TrackPort::BlockSourceTrackId() promise on teardown to avoid recent debug assert permafail.

[Triage Comment]
Cleans up some wpt assertion noise. Approved for 66.0b7.

Attachment #9041249 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Whiteboard: [stockwell disable-recommended] → [stockwell fixed]
Flags: needinfo?(james)
Attachment #9041460 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.