Closed Bug 1523795 Opened 7 months ago Closed 6 months ago

Perma Assertion failure: get() (dereferencing a UniquePtr containing nullptr), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:302

Categories

(Core :: WebRTC, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla67
Tracking Status
firefox67 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: bwc)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell disable-recommended])

Attachments

(2 files)

#[markdown(off)]
Filed by: shindli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=224821017&repo=mozilla-central

https://queue.taskcluster.net/v1/task/DjOt7sjQS-SoO5qyBt59pw/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/DjOt7sjQS-SoO5qyBt59pw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2019-01-29T22:57:56.307Z] 22:57:56 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/tests/crashtests/812785.html | 641 / 3672 (17%)
[task 2019-01-29T22:57:56.323Z] 22:57:56 INFO - ++DOMWINDOW == 44 (0x7fccf3239000) [pid = 1156] [serial = 3184] [outer = 0x7fccf8614000]
[task 2019-01-29T22:57:56.561Z] 22:57:56 INFO - [Child 1156, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /builds/worker/workspace/build/src/dom/base/nsContentUtils.cpp, line 3766
[task 2019-01-29T22:57:56.661Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:337: PeerConnectionImpl: PeerConnectionImpl constructor for
[task 2019-01-29T22:57:56.661Z] 22:57:56 INFO - [Parent 1008: Socket Thread]: D/mtransport NrIceCtx static call to find local stun addresses
[task 2019-01-29T22:57:56.665Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:337: PeerConnectionImpl: PeerConnectionImpl constructor for
[task 2019-01-29T22:57:56.669Z] 22:57:56 INFO - [Parent 1008: Socket Thread]: D/mtransport NrIceCtx static call to find local stun addresses
[task 2019-01-29T22:57:56.679Z] 22:57:56 INFO - [Child 1156: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2721: GetStats: Found no pipelines matching selector.
[task 2019-01-29T22:57:56.680Z] 22:57:56 INFO - [Child 1156: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2721: GetStats: Found no pipelines matching selector.
[task 2019-01-29T22:57:56.681Z] 22:57:56 INFO - Assertion failure: get() (dereferencing a UniquePtr containing nullptr), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:302
[task 2019-01-29T22:57:56.681Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionMedia] PeerConnectionMedia.cpp:51: OnProxyAvailable: Proxy Available: 0
[task 2019-01-29T22:57:56.682Z] 22:57:56 INFO - #01: libxul.so + 0x5921919
[task 2019-01-29T22:57:56.682Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.683Z] 22:57:56 INFO - #02: libxul.so + 0x594e753
[task 2019-01-29T22:57:56.683Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.683Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionMedia] PeerConnectionMedia.cpp:79: OnStunAddrsAvailable: receiving (5) stun addrs
[task 2019-01-29T22:57:56.684Z] 22:57:56 INFO - #03: libxul.so + 0x59599eb
[task 2019-01-29T22:57:56.684Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.685Z] 22:57:56 INFO - #04: libxul.so + 0x4c4b8df
[task 2019-01-29T22:57:56.685Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.686Z] 22:57:56 INFO - #05: libxul.so + 0x4c3ad4b
[task 2019-01-29T22:57:56.687Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.687Z] 22:57:56 INFO - #06: libxul.so + 0x4d9fff5
[task 2019-01-29T22:57:56.688Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.688Z] 22:57:56 INFO - #07: libxul.so + 0x4c4b8df
[task 2019-01-29T22:57:56.688Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.689Z] 22:57:56 INFO - #08: libxul.so + 0x4c3ad4b
[task 2019-01-29T22:57:56.689Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.689Z] 22:57:56 INFO - #09: libxul.so + 0x52ffae1
[task 2019-01-29T22:57:56.691Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.691Z] 22:57:56 INFO - #10: libxul.so + 0x52af403
[task 2019-01-29T22:57:56.691Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.692Z] 22:57:56 INFO - #11: libxul.so + 0x52af468
[task 2019-01-29T22:57:56.692Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.693Z] 22:57:56 INFO - #12: libxul.so + 0x4c3bd9e
[task 2019-01-29T22:57:56.693Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.814Z] 22:57:56 INFO - #13: _pt_root [nsprpub/pr/src/pthreads/ptthread.c:204]
[task 2019-01-29T22:57:56.815Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.815Z] 22:57:56 INFO - #14: libpthread.so.0 + 0x76ba
[task 2019-01-29T22:57:56.816Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.817Z] 22:57:56 INFO - #15: libc.so.6 + 0x10741d

Summary: Intermittent Assertion failure: get() (dereferencing a UniquePtr containing nullptr), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:302 → Perma Assertion failure: get() (dereferencing a UniquePtr containing nullptr), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:302
Flags: needinfo?(jib)

Odd, I don't know that we've landed anything webrtc recently. Symbols in the stack trace would have helped. I wonder why they're missing, this is a debug build.

A local build might help. Nico, would you mind taking a look? Looks stats-adjacent.

Flags: needinfo?(jib) → needinfo?(na-g)

Jan-Ivar, sure.

Flags: needinfo?(na-g)

It is not just a debug build, but a code coverage build. Code landed in PeerConnectionImpl.cpp in Bug 1494311, which is in that patch set. I am attempting to reproduce it locally.

I haven't been able to get a local ccov build (bug 1524116), or to get a 1-click loaner to work on a ccov build. I have however determined that the revision before Bug 1494311 does not exhibit the same behavior. It would be nice to get a stack trace. Can one run the mochitests using the artifacts produced by the ccov build?

Blocks: 1494311
Rank: 10
Priority: P5 → P1

So, I noticed that we were hitting this assertion while working on bug 1521879 when the socket process was enabled:

https://searchfox.org/mozilla-central/source/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c#412

It is possible that we're hitting this also in the CCov build, which was not running on my try pushes (because it is tier 2 I think). I'll look into that.

Ok, so it does not appear that the hypothesis in comment 7 is correct.

The lack of symbols is likely bug 1524051. Given that we are having no luck reproducing, I'm going to mark that as blocking for now.

Depends on: 1524051

Hmm, I see some failures on windows64 ccov in the latest comment, but those do not look related to me. Why were those categorized here?

Hi :bwc, sorry, it was my fault, fixed it.

(In reply to Nico Grunbaum [:ng] from comment #16)

Disassembling the libxul.so in the build artifact and looking at the addresses in some of the failures reveals, has so far revealed two failures: [1], [2].
[1] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#1841 On transceiver->()
[2] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#2997

So [1] isn't a UniquePtr, so that's a separate problem. As for [2], that is at least in a place that bug 1494311 touched, but it seems to use std::unique_ptr and not UniquePtr, which is weird.

(In reply to Byron Campen [:bwc] from comment #19)

(In reply to Nico Grunbaum [:ng] from comment #16)

Disassembling the libxul.so in the build artifact and looking at the addresses in some of the failures reveals, has so far revealed two failures: [1], [2].
[1] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#1841 On transceiver->()
[2] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#2997

So [1] isn't a UniquePtr, so that's a separate problem. As for [2], that is at least in a place that bug 1494311 touched, but it seems to use std::unique_ptr and not UniquePtr, which is weird.

In [2] query is a UniqPtr, this (595ca43) is the failure site (excerpt from the attached dump):

3761993- 595ca22:	e8 bb 9a 93 ff       	callq  52964e2 <RefPtr<mozilla::MediaTransportHandler>::operator->() const>
3761994- 595ca27:	48 89 c3             	mov    %rax,%rbx
3761995- 595ca2a:	48 8b 00             	mov    (%rax),%rax
3761996- 595ca2d:	4c 89 ff             	mov    %r15,%rdi
3761997- 595ca30:	48 ff 05 d1 8a b1 0e 	incq   0xeb18ad1(%rip)        # 14475508 <__gcov0._ZN7mozilla18PeerConnectionImpl19ExecuteStatsQuery_sEONS_9UniquePtrINS_13RTCStatsQueryENS_13DefaultDeleteIS2_EEEERK8nsTArrayI6RefPtrINS_13MediaPipelineEEERKS8_INS_21MediaTransportHandlerEE+0x528>
3761998- 595ca37:	4c 8b a8 80 00 00 00 	mov    0x80(%rax),%r13
3761999- 595ca3e:	e8 95 31 fd ff       	callq  592fbd8 <mozilla::UniquePtr<mozilla::RTCStatsQuery, mozilla::DefaultDelete<mozilla::RTCStatsQuery> >::operator->() const>
3762000: 595ca43:	48 8d 8d a0 fd ff ff 	lea    -0x260(%rbp),%rcx
3762001- 595ca4a:	48 ff 05 bf 8a b1 0e 	incq   0xeb18abf(%rip)        # 14475510 

That's pretty weird. Maybe we need to grab a copy of query->now before putting together that promise chain.

Duplicate of this bug: 1524621

I'm not seeing this failure on the try push in comment 28. I think this may be the fix we need.

Assignee: nobody → docfaraday
Pushed by bcampen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f2bb667f3031
Make temp copy of query->now to avoid access after query has been moved. r=ng
Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.