1523795 - Perma Assertion failure: get() (dereferencing a UniquePtr containing nullptr), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:302

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P1)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: shindli [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=224821017&repo=mozilla-central

https://queue.taskcluster.net/v1/task/DjOt7sjQS-SoO5qyBt59pw/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/DjOt7sjQS-SoO5qyBt59pw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2019-01-29T22:57:56.307Z] 22:57:56 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/tests/crashtests/812785.html | 641 / 3672 (17%)
[task 2019-01-29T22:57:56.323Z] 22:57:56 INFO - ++DOMWINDOW == 44 (0x7fccf3239000) [pid = 1156] [serial = 3184] [outer = 0x7fccf8614000]
[task 2019-01-29T22:57:56.561Z] 22:57:56 INFO - [Child 1156, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /builds/worker/workspace/build/src/dom/base/nsContentUtils.cpp, line 3766
[task 2019-01-29T22:57:56.661Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:337: PeerConnectionImpl: PeerConnectionImpl constructor for
[task 2019-01-29T22:57:56.661Z] 22:57:56 INFO - [Parent 1008: Socket Thread]: D/mtransport NrIceCtx static call to find local stun addresses
[task 2019-01-29T22:57:56.665Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:337: PeerConnectionImpl: PeerConnectionImpl constructor for
[task 2019-01-29T22:57:56.669Z] 22:57:56 INFO - [Parent 1008: Socket Thread]: D/mtransport NrIceCtx static call to find local stun addresses
[task 2019-01-29T22:57:56.679Z] 22:57:56 INFO - [Child 1156: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2721: GetStats: Found no pipelines matching selector.
[task 2019-01-29T22:57:56.680Z] 22:57:56 INFO - [Child 1156: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2721: GetStats: Found no pipelines matching selector.
[task 2019-01-29T22:57:56.681Z] 22:57:56 INFO - Assertion failure: get() (dereferencing a UniquePtr containing nullptr), at /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:302
[task 2019-01-29T22:57:56.681Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionMedia] PeerConnectionMedia.cpp:51: OnProxyAvailable: Proxy Available: 0
[task 2019-01-29T22:57:56.682Z] 22:57:56 INFO - #01: libxul.so + 0x5921919
[task 2019-01-29T22:57:56.682Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.683Z] 22:57:56 INFO - #02: libxul.so + 0x594e753
[task 2019-01-29T22:57:56.683Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.683Z] 22:57:56 INFO - [Child 1156: Main Thread]: I/signaling [main|PeerConnectionMedia] PeerConnectionMedia.cpp:79: OnStunAddrsAvailable: receiving (5) stun addrs
[task 2019-01-29T22:57:56.684Z] 22:57:56 INFO - #03: libxul.so + 0x59599eb
[task 2019-01-29T22:57:56.684Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.685Z] 22:57:56 INFO - #04: libxul.so + 0x4c4b8df
[task 2019-01-29T22:57:56.685Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.686Z] 22:57:56 INFO - #05: libxul.so + 0x4c3ad4b
[task 2019-01-29T22:57:56.687Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.687Z] 22:57:56 INFO - #06: libxul.so + 0x4d9fff5
[task 2019-01-29T22:57:56.688Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.688Z] 22:57:56 INFO - #07: libxul.so + 0x4c4b8df
[task 2019-01-29T22:57:56.688Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.689Z] 22:57:56 INFO - #08: libxul.so + 0x4c3ad4b
[task 2019-01-29T22:57:56.689Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.689Z] 22:57:56 INFO - #09: libxul.so + 0x52ffae1
[task 2019-01-29T22:57:56.691Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.691Z] 22:57:56 INFO - #10: libxul.so + 0x52af403
[task 2019-01-29T22:57:56.691Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.692Z] 22:57:56 INFO - #11: libxul.so + 0x52af468
[task 2019-01-29T22:57:56.692Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.693Z] 22:57:56 INFO - #12: libxul.so + 0x4c3bd9e
[task 2019-01-29T22:57:56.693Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.814Z] 22:57:56 INFO - #13: _pt_root [nsprpub/pr/src/pthreads/ptthread.c:204]
[task 2019-01-29T22:57:56.815Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.815Z] 22:57:56 INFO - #14: libpthread.so.0 + 0x76ba
[task 2019-01-29T22:57:56.816Z] 22:57:56 INFO -
[task 2019-01-29T22:57:56.817Z] 22:57:56 INFO - #15: libc.so.6 + 0x10741d

Comment 1

[Daniel Varga [:dvarga]]

Started from: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=e54fff25332417ae1cf18b933882798a33d24ec9&searchStr=mda3&selectedJob=225029602

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Odd, I don't know that we've landed anything webrtc recently. Symbols in the stack trace would have helped. I wonder why they're missing, this is a debug build.

A local build might help. Nico, would you mind taking a look? Looks stats-adjacent.

Comment 3

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Jan-Ivar, sure.

Comment 4

[Nico Grunbaum [:ng, @chew:mozilla.org]]

It is not just a debug build, but a code coverage build. Code landed in PeerConnectionImpl.cpp in Bug 1494311, which is in that patch set. I am attempting to reproduce it locally.

Comment 5

[Intermittent Failures Robot]

72 failures in 481 pushes (0.15 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 72

Platform breakdown:

• windows10-64-ccov: 26
• linux64-ccov: 46

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-01-30&endday=2019-01-30&tree=all

Comment 6

[Nico Grunbaum [:ng, @chew:mozilla.org]]

I haven't been able to get a local ccov build (bug 1524116), or to get a 1-click loaner to work on a ccov build. I have however determined that the revision before Bug 1494311 does not exhibit the same behavior. It would be nice to get a stack trace. Can one run the mochitests using the artifacts produced by the ccov build?

Comment 7

[Byron Campen [:bwc]]

So, I noticed that we were hitting this assertion while working on bug 1521879 when the socket process was enabled:

https://searchfox.org/mozilla-central/source/media/mtransport/third_party/nICEr/src/ice/ice_media_stream.c#412

It is possible that we're hitting this also in the CCov build, which was not running on my try pushes (because it is tier 2 I think). I'll look into that.

Comment 8

[Byron Campen [:bwc]]

Ok, so it does not appear that the hypothesis in comment 7 is correct.

The lack of symbols is likely bug 1524051. Given that we are having no luck reproducing, I'm going to mark that as blocking for now.

Comment 9

[Intermittent Failures Robot]

17 failures in 517 pushes (0.033 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 17

Platform breakdown:

• linux64-ccov: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-01-31&endday=2019-01-31&tree=all

Comment 10

[Intermittent Failures Robot]

42 failures in 481 pushes (0.087 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 42

Platform breakdown:

• linux64-ccov: 42

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-01&endday=2019-02-01&tree=all

Comment 11

[Intermittent Failures Robot]

25 failures in 176 pushes (0.142 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 25

Platform breakdown:

• linux64-ccov: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-02&endday=2019-02-02&tree=all

Comment 12

[Intermittent Failures Robot]

200 failures in 2628 pushes (0.076 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 200

Platform breakdown:

• windows10-64-ccov: 26
• linux64-ccov: 174

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-01-28&endday=2019-02-03&tree=all

Comment 13

[Byron Campen [:bwc]]

Hmm, I see some failures on windows64 ccov in the latest comment, but those do not look related to me. Why were those categorized here?

Comment 14

[Daniel Varga [:dvarga]]

Hi :bwc, sorry, it was my fault, fixed it.

Comment 15

[Intermittent Failures Robot]

51 failures in 379 pushes (0.135 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 51

Platform breakdown:

• linux64-ccov: 51

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-04&endday=2019-02-04&tree=all

Comment 16

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Disassembling the libxul.so in the build artifact and looking at the addresses in some of the failures reveals, has so far revealed two failures: [1], [2].
[1] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#1841 On transceiver->()
[2] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#2997

Comment 17

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Attachment: bug-1523795-second-failure-at-595ca43.txt

Comment 18

[Intermittent Failures Robot]

40 failures in 442 pushes (0.09 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 40

Platform breakdown:

• linux64-ccov: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-05&endday=2019-02-05&tree=all

Comment 19

[Byron Campen [:bwc]]

(In reply to Nico Grunbaum [:ng] from comment #16)

Disassembling the libxul.so in the build artifact and looking at the addresses in some of the failures reveals, has so far revealed two failures: [1], [2].
[1] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#1841 On transceiver->()
[2] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#2997

So [1] isn't a UniquePtr, so that's a separate problem. As for [2], that is at least in a place that bug 1494311 touched, but it seems to use std::unique_ptr and not UniquePtr, which is weird.

Comment 20

[Nico Grunbaum [:ng, @chew:mozilla.org]]

(In reply to Byron Campen [:bwc] from comment #19)

(In reply to Nico Grunbaum [:ng] from comment #16)

Disassembling the libxul.so in the build artifact and looking at the addresses in some of the failures reveals, has so far revealed two failures: [1], [2].
[1] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#1841 On transceiver->()
[2] https://searchfox.org/mozilla-central/rev/9eb30227b21e0aa40d51d9f9b08bb0b113c5fadb/media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp#2997

So [1] isn't a UniquePtr, so that's a separate problem. As for [2], that is at least in a place that bug 1494311 touched, but it seems to use std::unique_ptr and not UniquePtr, which is weird.

In [2] query is a UniqPtr, this (595ca43) is the failure site (excerpt from the attached dump):

3761993- 595ca22:	e8 bb 9a 93 ff       	callq  52964e2 <RefPtr<mozilla::MediaTransportHandler>::operator->() const>
3761994- 595ca27:	48 89 c3             	mov    %rax,%rbx
3761995- 595ca2a:	48 8b 00             	mov    (%rax),%rax
3761996- 595ca2d:	4c 89 ff             	mov    %r15,%rdi
3761997- 595ca30:	48 ff 05 d1 8a b1 0e 	incq   0xeb18ad1(%rip)        # 14475508 <__gcov0._ZN7mozilla18PeerConnectionImpl19ExecuteStatsQuery_sEONS_9UniquePtrINS_13RTCStatsQueryENS_13DefaultDeleteIS2_EEEERK8nsTArrayI6RefPtrINS_13MediaPipelineEEERKS8_INS_21MediaTransportHandlerEE+0x528>
3761998- 595ca37:	4c 8b a8 80 00 00 00 	mov    0x80(%rax),%r13
3761999- 595ca3e:	e8 95 31 fd ff       	callq  592fbd8 <mozilla::UniquePtr<mozilla::RTCStatsQuery, mozilla::DefaultDelete<mozilla::RTCStatsQuery> >::operator->() const>
3762000: 595ca43:	48 8d 8d a0 fd ff ff 	lea    -0x260(%rbp),%rcx
3762001- 595ca4a:	48 ff 05 bf 8a b1 0e 	incq   0xeb18abf(%rip)        # 14475510 

Comment 21

[Byron Campen [:bwc]]

That's pretty weird. Maybe we need to grab a copy of query->now before putting together that promise chain.

Comment 22

[Intermittent Failures Robot]

36 failures in 478 pushes (0.075 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 36

Platform breakdown:

• linux64-ccov: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-06&endday=2019-02-06&tree=all

Comment 23

[Intermittent Failures Robot]

38 failures in 486 pushes (0.078 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 38

Platform breakdown:

• linux64-ccov: 38

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-07&endday=2019-02-07&tree=all

Comment 25

[Intermittent Failures Robot]

47 failures in 550 pushes (0.085 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 47

Platform breakdown:

• linux64-ccov: 47

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-08&endday=2019-02-08&tree=all

Comment 26

[Intermittent Failures Robot]

50 failures in 204 pushes (0.245 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 50

Platform breakdown:

• linux64-ccov: 50

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-09&endday=2019-02-09&tree=all

Comment 27

[Intermittent Failures Robot]

322 failures in 2658 pushes (0.121 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• try: 12
• mozilla-central: 310

Platform breakdown:

• linux64-ccov: 322

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 28

[Byron Campen [:bwc]]

Experimental try push here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ac2e5e62212cfa6e6a7a46680cfdae9778fc808

Comment 29

[Byron Campen [:bwc]]

I'm not seeing this failure on the try push in comment 28. I think this may be the fix we need.

Comment 30

[Byron Campen [:bwc]]

Attachment: Bug 1523795: Make temp copy of query->now to avoid access after query has been moved. r?ng

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2bb667f3031
Make temp copy of query->now to avoid access after query has been moved. r=ng

Comment 32

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/f2bb667f3031

Comment 33

[Intermittent Failures Robot]

16 failures in 533 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 16

Platform breakdown:

• linux64-ccov: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-11&endday=2019-02-11&tree=all

Comment 34

[Intermittent Failures Robot]

16 failures in 2832 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform breakdown:

• linux64-ccov: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1523795&startday=2019-02-11&endday=2019-02-17&tree=all