851307 - Assertion failure and crash on shutdown due to use of SocketTransportService after it's shutdown: ((bool)(__builtin_expect(!!(!NS_FAILED_impl(rv)), 1))), at ../../../dist/include/mtransport/runnable_utils.h:48 [@ mozilla::DataChannelConnection::Release()]

Status: RESOLVED FIXED

(Core :: WebRTC: Networking, defect, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: testcase

+++ This bug was initially created as a clone of Bug #850773 +++

One more of those builtin assertions but this time with datachannel. The patch attached is a simple conversion of the datachannel test of webrtc-landing.

0:37.60 Crash reason:  EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
 0:37.60 Crash address: 0x0
 0:37.60 
 0:37.60 Thread 0 (crashed)
 0:37.60 0  XUL!_ZN7mozillaL13RUN_ON_THREADEP14nsIEventTargetP11nsIRunnablej + 0x9f
 0:37.60 rbx = 0x00007fff7500d630   r12 = 0x0000000108023ce8
 0:37.60 r13 = 0x000000011ae37c40   r14 = 0x000000010013c998
 0:37.60 r15 = 0x00000001191c8fb0   rip = 0x000000010121ac0f
 0:37.60 rsp = 0x00007fff5fbfe800   rbp = 0x00007fff5fbfe820
 0:37.60 Found by: given as instruction pointer in context
 0:37.60 1  XUL!mozilla::DataChannelConnection::Release() [DataChannel.cpp:47a5f6fb1e15 : 238 + 0x7]
 0:37.60 rip = 0x000000010121b0a0   rsp = 0x00007fff5fbfe810
 0:37.60 rbp = 0x00007fff5fbfe820
 0:37.60 Found by: stack scanning
 0:37.60 2  XUL!mozilla::DataChannelConnection::Destroy() [DataChannel.cpp:47a5f6fb1e15 : 229 + 0xa]
 0:37.60 rip = 0x000000010121adf8   rsp = 0x00007fff5fbfe830
 0:37.60 rbp = 0x00007fff5fbfe850
 0:37.60 Found by: stack scanning
 0:37.60 3  XUL!sipcc::PeerConnectionImpl::CloseInt(bool) [PeerConnectionImpl.cpp:47a5f6fb1e15 : 1165 + 0x4]
 0:37.60 rip = 0x00000001030d25f1   rsp = 0x00007fff5fbfe860
 0:37.60 rbp = 0x00007fff5fbfe880
 0:37.60 Found by: stack scanning
 0:37.60 4  XUL!sipcc::PeerConnectionImpl::~PeerConnectionImpl() [PeerConnectionImpl.cpp:47a5f6fb1e15 : 260 + 0x9]
 0:37.60 rip = 0x00000001030d2269   rsp = 0x00007fff5fbfe890
 0:37.60 rbp = 0x00007fff5fbfe8e0
 0:37.60 Found by: stack scanning
 0:37.60 5  libsystem_c.dylib + 0x4d6aa
 0:37.60 rip = 0x00007fff8d4056ab   rsp = 0x00007fff5fbfe8b0
 0:37.60 rbp = 0x00007fff5fbfe8e0
 0:37.60 Found by: stack scanning
 0:37.60 6  XUL!sipcc::PeerConnectionImpl::~PeerConnectionImpl() [PeerConnectionImpl.cpp:47a5f6fb1e15 : 251 + 0x4]
 0:37.60 rip = 0x00000001030d20ae   rsp = 0x00007fff5fbfe8f0
 0:37.60 rbp = 0x00007fff5fbfe900
 0:37.60 Found by: stack scanning
 0:37.61 7  XUL!sipcc::PeerConnectionImpl::Release() [PeerConnectionImpl.cpp:47a5f6fb1e15 : 231 + 0xb]
 0:37.61 rip = 0x00000001030d1d7f   rsp = 0x00007fff5fbfe910
 0:37.61 rbp = 0x00007fff5fbfe920
 0:37.61 Found by: stack scanning
 0:37.61 8  XUL!XPCJSRuntime::GCCallback(JSRuntime*, JSGCStatus) [XPCJSRuntime.cpp:47a5f6fb1e15 : 595 + 0x9]
 0:37.61 rip = 0x000000010210df57   rsp = 0x00007fff5fbfe930
 0:37.61 rbp = 0x00007fff5fbfe970
 0:37.61 Found by: stack scanning
 0:37.61 9  XUL!PRMJ_Now() [prmjtime.cpp:47a5f6fb1e15 : 204 + 0x4]
 0:37.61 rip = 0x000000010337e3f3   rsp = 0x00007fff5fbfe940
 0:37.61 rbp = 0x00007fff5fbfe970
 0:37.61 Found by: stack scanning
 0:37.61 10  XUL!Collect [jsgc.cpp:47a5f6fb1e15 : 4507 + 0x9]
 0:37.61 rip = 0x000000010324b658   rsp = 0x00007fff5fbfe980
 0:37.61 rbp = 0x00007fff5fbfe9e0
 0:37.61 Found by: stack scanning
 0:37.61 11  XUL!nsXREDirProvider::DoShutdown() [nsXREDirProvider.cpp:47a5f6fb1e15 : 865 + 0x4]
 0:37.61 rip = 0x000000010101429e   rsp = 0x00007fff5fbfe9f0
 0:37.61 rbp = 0x00007fff5fbfea20
 0:37.61 Found by: stack scanning
 0:37.61 12  XUL!ScopedXPCOMStartup::~ScopedXPCOMStartup() [nsAppRunner.cpp:47a5f6fb1e15 : 1117 + 0xe]
 0:37.61 rip = 0x000000010100560d   rsp = 0x00007fff5fbfea30
 0:37.61 rbp = 0x00007fff5fbfea60
 0:37.61 Found by: stack scanning
 0:37.61 13  XUL!XREMain::XRE_main(int, char**, nsXREAppData const*) [nsAppRunner.cpp:47a5f6fb1e15 : 1103 + 0x7]
 0:37.61 rip = 0x000000010100c56e   rsp = 0x00007fff5fbfea70
 0:37.61 rbp = 0x00007fff5fbfeab0
 0:37.61 Found by: stack scanning
 0:37.61 14  XUL!nsXREDirProvider::nsXREDirProvider() [nsTArray-inl.h : 14 + 0xc]
 0:37.61 rip = 0x000000010100fefb   rsp = 0x00007fff5fbfea90
 0:37.61 rbp = 0x00007fff5fbfeab0
 0:37.61 Found by: stack scanning
 0:37.61 15  XUL!XRE_main [nsAppRunner.cpp:47a5f6fb1e15 : 4150 + 0x10]
 0:37.61 rip = 0x000000010100c88a   rsp = 0x00007fff5fbfeac0
 0:37.61 rbp = 0x00007fff5fbfec40
 0:37.61

Comment 1

[Jason Smith [:jsmith]]

Basically, anything that's going to block the ability to run data channel mochitests in CI is automatic blocker.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

It's unfortunate that we don't have the mochitest output from this, in that clearly an assertion fired but it's not 100% clear which one.  

Also, installing this patch and trying to run the mochitests (in particular running it as the only mochitest) never seems to exit and gets errors thrown about missing JS functions.  Perhaps there's an unspecified supporting framework patch needed?

Without more information or a reproducible case it will be hard to make any headway on understanding this.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The failing disconnect method isn't a problem. It was a function I have removed. Please ignore it. So here the last lines for the mochitest output before I hit Cmd+Q to quit Firefox.

0:20.36 WARNING: NS_ENSURE_TRUE(thread) failed: file /Volumes/data/code/firefox/nightly/netwerk/base/src/nsSocketTransportService2.cpp, line 115
 0:20.36 319754240[1080add20]: [GSM Task|def] fsm.c:346: SIPCC-FSM: 2/2, fsm_change_state: DEF: CONNECTED -> IDLE
 0:20.36 319754240[1080add20]: [GSM Task|def] dcsm.c:397: SIPCC-DCSM: dcsm_update_gsm_state: 2 : DCSM_READY --> DCSM_READY
 0:20.36 294600704[1080ad4f0]: [CCAPP Task|def] ccprovider.c:1502: SIPCC-SIP_CC_PROV: ccappUpdateSessionData: Call_STATE:. state=1, attr=0, cause=5, instance=0
 0:20.36 319754240[1080add20]: [GSM Task|fsm_sm] fsmdef.c:1084: SIPCC-FSM: 0/0, fsmdef_init_dcb: call_not_counted_in_mnc_bt = FALSE
 0:20.36 319754240[1080add20]: [GSM Task|def] fsm.c:346: SIPCC-FSM: 0/2, fsm_change_state: DEF: IDLE -> IDLE
 0:20.36 319754240[1080add20]: [GSM Task|def] dcsm.c:397: SIPCC-DCSM: dcsm_update_gsm_state: 2 : DCSM_READY --> DCSM_READY
 0:20.36 WARNING: NS_ENSURE_TRUE(thread) failed: file /Volumes/data/code/firefox/nightly/netwerk/base/src/nsSocketTransportService2.cpp, line 129
 0:20.36 Assertion failure: ((bool)(__builtin_expect(!!(!NS_FAILED_impl(rv)), 1))), at ../../../dist/include/mtransport/runnable_utils.h:48

It looks like an assertion already failed before in nsSocketTransportService2.cpp.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Aha.  Ok, so it appears that something tried to dispatch to the STS thread after it had shut down.  From then, all bets are likely off and this follow-on assertion makes sense.

I'm *hoping* this is due to the re-entrancy problem.

Comment 5

[bhavana bajaj [:bajaj]]

Is FF21 affected ?

Comment 6

[Eric Rescorla (:ekr)]

can you please retest, henrik?

Comment 7

[Maire Reavy [:mreavy]]

We believe this bug was fixed when the patch for bug 844493 landed.  Jan-Ivar is retesting Bug 842749 for the same reason.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I don't see this anymore until now. Lets close it given the last comment. We can reopen if it happens again. Otherwise assume for now it's fixed.

Comment 9

[Jason Smith [:jsmith]]

Will be covered by the work going on for basic smoke test data channel automation. Don't need to track this.