Closed Bug 848966 Opened 7 years ago Closed 7 years ago

Intermittent test_peerConnection_basicAudioVideo.html | Exited with code -1073741819 during test run | application crashed [@ nsRunnable::Release()] [@ nsRefPtr<nsIDOMEventListener>::~nsRefPtr<nsIDOMEventListener>()]

Categories

(Core :: WebRTC: Signaling, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla22

People

(Reporter: RyanVM, Unassigned)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [webrtc][blocking-webrtc+][qa-])

Crash Data

Going out on a limb that this is related to bug 848423. This is only on inbound (so far) and all the instances are from after it landed.

https://tbpl.mozilla.org/php/getParsedLog.php?id=20432737&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=20422286&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=20427264&tree=Mozilla-Inbound

Rev3 WINNT 5.1 mozilla-inbound pgo test mochitest-2 on 2013-03-07 12:50:47 PST for push 908a5ff75e5e
slave: talos-r3-xp-023

13:12:57     INFO -  PROCESS-CRASH | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideo.html | application crashed [@ nsRunnable::Release()]
13:12:57     INFO -  Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmpuucmmz\minidumps\4b38ac37-b53d-4459-912e-402812a31329.dmp
13:12:57     INFO -  Operating system: Windows NT
13:12:57     INFO -                    5.1.2600 Service Pack 2
13:12:57     INFO -  CPU: x86
13:12:57     INFO -       GenuineIntel family 6 model 23 stepping 10
13:12:57     INFO -       2 CPUs
13:12:57     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_WRITE
13:12:57     INFO -  Crash address: 0x5
13:12:57     INFO -  Thread 6 (crashed)
13:12:57     INFO -   0  xul.dll!nsRunnable::Release() [nsThreadUtils.cpp:908a5ff75e5e : 31 + 0xd]
13:12:57     INFO -      eip = 0x015723bd   esp = 0x03bcfdd4   ebp = 0x03bcfdd8   ebx = 0x013fd101
13:12:57     INFO -      esi = 0x00000001   edi = 0x00000001   eax = 0xffffffff   ecx = 0x06fae910
13:12:57     INFO -      edx = 0x00000005   efl = 0x00010286
13:12:57     INFO -      Found by: given as instruction pointer in context
13:12:57     INFO -   1  xul.dll!mozilla::TransportFlow::Release() [transportflow.h:908a5ff75e5e : 65 + 0x21]
13:12:57     INFO -      eip = 0x01c1a0ee   esp = 0x03bcfde0   ebp = 0x03bcfde8
13:12:57     INFO -      Found by: call frame info
13:12:57     INFO -   2  xul.dll!nsRefPtr<mozilla::TransportFlow>::~nsRefPtr<mozilla::TransportFlow>() [nsAutoPtr.h:908a5ff75e5e : 880 + 0x5]
13:12:57     INFO -      eip = 0x01c7345e   esp = 0x03bcfdf0   ebp = 0x03bcfdfc
13:12:57     INFO -      Found by: call frame info
13:12:57     INFO -   3  xul.dll!mozilla::runnable_args_nm_1<void (*)(nsRefPtr<mozilla::TransportFlow>),already_AddRefed<mozilla::TransportFlow> >::Run() [runnable_utils_generated.h:908a5ff75e5e : 85 + 0xc]
13:12:57     INFO -      eip = 0x01cbaa3b   esp = 0x03bcfdf8   ebp = 0x03bcfdfc
13:12:57     INFO -      Found by: call frame info
13:12:57     INFO -   4  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:908a5ff75e5e : 627 + 0x13]
13:12:57     INFO -      eip = 0x0157a596   esp = 0x03bcfe04   ebp = 0x03bcfe70
13:12:57     INFO -      Found by: call frame info
13:12:57     INFO -   5  xul.dll!NS_ProcessNextEvent_P(nsIThread *,bool) [nsThreadUtils.cpp:908a5ff75e5e : 238 + 0x1e]
13:12:57     INFO -      eip = 0x015563bd   esp = 0x03bcfe78   ebp = 0x03bcfe84
13:12:57     INFO -      Found by: call frame info
13:12:57     INFO -   6  xul.dll!nsSocketTransportService::Run() [nsSocketTransportService2.cpp:908a5ff75e5e : 649 + 0x7]
13:12:57     INFO -      eip = 0x0152bd08   esp = 0x03bcfe8c   ebp = 0x03bcfea8
13:12:57     INFO -      Found by: call frame info

07:45:17     INFO -  PROCESS-CRASH | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudioVideoCombined.html | application crashed [@ nsRefPtr<nsIDOMEventListener>::~nsRefPtr<nsIDOMEventListener>()]
07:45:17     INFO -  Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmpqg98xo\minidumps\e8c8ddb4-4b0b-4af1-b135-32497e88e362.dmp
07:45:17     INFO -  Operating system: Windows NT
07:45:17     INFO -                    5.1.2600 Service Pack 2
07:45:17     INFO -  CPU: x86
07:45:17     INFO -       GenuineIntel family 6 model 23 stepping 10
07:45:17     INFO -       2 CPUs
07:45:17     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
07:45:17     INFO -  Crash address: 0x4c191008
07:45:17     INFO -  Thread 0 (crashed)
07:45:17     INFO -   0  xul.dll!nsRefPtr<nsIDOMEventListener>::~nsRefPtr<nsIDOMEventListener>() [nsAutoPtr.h:1768185637fd : 880 + 0x2]
07:45:17     INFO -      eip = 0x01533868   esp = 0x0012e444   ebp = 0x0012e454   ebx = 0x0c4aa840
07:45:17     INFO -      esi = 0x0c4aa860   edi = 0x00000001   eax = 0x0998f5ff   ecx = 0x4c191000
07:45:17     INFO -      edx = 0x80000008   efl = 0x00210206
07:45:17     INFO -      Found by: given as instruction pointer in context
07:45:17     INFO -   1  xul.dll!UndisplayedNode::`scalar deleting destructor'(unsigned int) + 0x35
07:45:17     INFO -      eip = 0x01428e64   esp = 0x0012e448   ebp = 0x0012e454
07:45:17     INFO -      Found by: call frame info
07:45:17     INFO -   2  xul.dll!RemoveUndisplayedEntry [nsFrameManager.cpp:1768185637fd : 2005 + 0x29]
07:45:17     INFO -      eip = 0x01456641   esp = 0x0012e45c   ebp = 0x0012e470
07:45:17     INFO -      Found by: call frame info
07:45:17     INFO -   3  plds4.dll!PL_HashTableEnumerateEntries [plhash.c:1768185637fd : 374 + 0xa]
07:45:17     INFO -      eip = 0x003e16dd   esp = 0x0012e478   ebp = 0x0bf35380
07:45:17     INFO -      Found by: call frame info
07:45:17     INFO -   4  xul.dll!nsFrameManager::Destroy() [nsFrameManager.cpp:1768185637fd : 224 + 0x17]
07:45:17     INFO -      eip = 0x0140ee0c   esp = 0x0012e4a4   ebp = 0x0012e4c4
07:45:17     INFO -      Found by: call frame info
07:45:17     INFO -   5  xul.dll!nsCSSFrameConstructor::WillDestroyFrameTree() [nsCSSFrameConstructor.cpp:1768185637fd : 8696 + 0x6]
07:45:17     INFO -      eip = 0x015d6e71   esp = 0x0012e4bc   ebp = 0x0012e4c4
07:45:17     INFO -      Found by: call frame info
07:45:17     INFO -   6  xul.dll!PresShell::Destroy() [nsPresShell.cpp:1768185637fd : 1086 + 0x7]
07:45:17     INFO -      eip = 0x01447dbc   esp = 0x0012e4cc   ebp = 0x0012e4e4
07:45:17     INFO -      Found by: call frame info
07:45:17     INFO -   7  xul.dll!nsDocumentViewer::DestroyPresShell() [nsDocumentViewer.cpp:1768185637fd : 4366 + 0x6]
07:45:17     INFO -      eip = 0x01586347   esp = 0x0012e4ec   ebp = 0x0012e500
07:45:17     INFO -      Found by: call frame info

10:41:02     INFO -  PROCESS-CRASH | /tests/dom/media/tests/mochitest/test_peerConnection_basicVideo.html | application crashed [@ mozilla::MediaStreamGraphImpl::ExtractPendingInput(mozilla::SourceMediaStream *,__int64,bool *)]
10:41:02     INFO -  Crash dump filename: c:\docume~1\cltbld\locals~1\temp\tmp4xf3v0\minidumps\cfb31ecf-da3e-43db-a569-564602a175bb.dmp
10:41:02     INFO -  Operating system: Windows NT
10:41:02     INFO -                    5.1.2600 Service Pack 2
10:41:02     INFO -  CPU: x86
10:41:02     INFO -       GenuineIntel family 6 model 23 stepping 10
10:41:02     INFO -       2 CPUs
10:41:02     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
10:41:02     INFO -  Crash address: 0x8
10:41:02     INFO -  Thread 43 (crashed)
10:41:02     INFO -   0  xul.dll!mozilla::MediaStreamGraphImpl::ExtractPendingInput(mozilla::SourceMediaStream *,__int64,bool *) [MediaStreamGraph.cpp:814a0c94b215 : 141 + 0x25]
10:41:02     INFO -      eip = 0x02071ada   esp = 0x09fffdd8   ebp = 0x09fffdfc   ebx = 0x058fe110
10:41:02     INFO -      esi = 0x0ab29fa8   edi = 0x093121c0   eax = 0x00000000   ecx = 0x0ac6839c
10:41:02     INFO -      edx = 0x00000001   efl = 0x00010246
10:41:02     INFO -      Found by: given as instruction pointer in context
10:41:02     INFO -   1  xul.dll!mozilla::MediaStreamGraphImpl::RunThread() [MediaStreamGraph.cpp:814a0c94b215 : 956 + 0x13]
10:41:02     INFO -      eip = 0x02162dd6   esp = 0x09fffe04   ebp = 0x09fffe9c
10:41:02     INFO -      Found by: call frame info
10:41:02     INFO -   2  xul.dll!mozilla::`anonymous namespace'::MediaStreamGraphThreadRunnable::Run() [MediaStreamGraph.cpp:814a0c94b215 : 1104 + 0xa]
10:41:02     INFO -      eip = 0x02163504   esp = 0x09fffea4   ebp = 0x09fffea8
10:41:02     INFO -      Found by: call frame info
10:41:02     INFO -   3  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:814a0c94b215 : 627 + 0x13]
10:41:02     INFO -      eip = 0x014de4f6   esp = 0x09fffeb0   ebp = 0x09ffff1c
10:41:02     INFO -      Found by: call frame info
10:41:02     INFO -   4  xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:814a0c94b215 : 265 + 0x1b]
10:41:02     INFO -      eip = 0x0157f247   esp = 0x09ffff24   ebp = 0x09ffff44
10:41:02     INFO -      Found by: call frame info
10:41:02     INFO -   5  nspr4.dll!_PR_NativeRunThread [pruthr.c:814a0c94b215 : 395 + 0x8]
10:41:02     INFO -      eip = 0x003a2f4f   esp = 0x09ffff4c   ebp = 0x09ffff68
10:41:02     INFO -      Found by: call frame info
10:41:02     INFO -   6  nspr4.dll!pr_root [w95thred.c:814a0c94b215 : 90 + 0xc]
10:41:02     INFO -      eip = 0x003a44ed   esp = 0x09ffff70   ebp = 0x09ffffa8
10:41:02     INFO -      Found by: call frame info
All PGO builds too, goodie.
Bug 848423 backed out for now. I'll resolve this once I can confirm with PGO builds that this is gone.
https://hg.mozilla.org/integration/mozilla-inbound/rev/1d028bd354d0
https://tbpl.mozilla.org/php/getParsedLog.php?id=20427264&tree=Mozilla-Inbound is from before bug 848423 landed (and stuck), so I guess that makes it unlikely to be the cause.
OS: Windows XP → All
Further from RyanVM: "My guess is that whatever broke it is in here somewhere: https://hg.mozilla.org/mozilla-central/pushloghtml?startID=24405&endID=24406"
Okay, a search through m-i for changes in this range that impacted files in the media subtree narrows us down:

Bug 839677 - Remove problematic gsm_msg_queue and use gsm_msgq instead
Bug 828917 - move critsect to cover thread existence test (upstream issue 1465) r=derf
Bug 837035 - Support DataChannels with SDP Negotiation r=jesup
Bug 836824 - Make libsoundtouch build on FreeBSD and Mac without command-line tools. r=khuey

The following changesets were backed out, so they aren't suspects:
Bug 848423 - Change nearly all WebRTC instances of linked_ptr<> to nsRefPtr<> r=ehugg
Bug 839650 - Add debugs to MediaStreamGraph to ease investigation of issues in the future r=roc
A quick rundown of the suspect bugs:


Bug 839677: This patch simply takes two variables that were the same value and collapses them to one variable. It is almost certainly not the cause.

Bug 828917: Jesup has indicated that he "can guarantee" it's not this bug.

Bug 837035: Given the number of changes in here, it's *possible* that it could be the root; however, I don't see anything specific to call termination that would be particularly suspect.

Bug 836824: This rather small patch would seem to only impact CPU feature detection.

With this list of suspects, I'd start with Bug 837035 as the most likely candidate.
The wild set of random crash locations speaks to some sort of memory error whomping different things.  An asan run wouldn't be a bad idea, though the interaction with PGO could be a problem.  How frequent/easy is it to trigger?  Is it amenable to a small amount of bisection? (painful on PGO, true)
Seems to be near-perma fail on the Windows PGO builds.
So, backing out bug 837035, bug 828917, and bug 839677 did indeed make this go away. Bug 837035 was the first backout and the Win PGO M2 runs were clean. I will reland bug 828917 and bug 839677 in the morning when I can trigger PGO on them and report back.
Fixed by backing out bug 837035.
Blocks: 837035
No longer blocks: 848423
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla22
Whiteboard: [webrtc][blocking-webrtc+] → [webrtc][blocking-webrtc+][qa-]
Flags: in-testsuite+
You need to log in before you can comment on or make changes to this bug.