657588 - [adbe 2875277] Crash [@ std::_Construct<MessageLoop::PendingTask, MessageLoop::PendingTask>(MessageLoop::PendingTask*, MessageLoop::PendingTask const&) ] | ABORT: CRT ASSERT dbgheap.c(1279) : Assertion failed: _CrtIsValidHeapPointer(pUserData)

Status: VERIFIED FIXED

(Core Graveyard :: Plug-ins, defect)

Description

[Bob Clary [:bc] (inactive)]

Tested with 10.3.181.14

Crash 1
1. http://www.myspace.com/plies
2. Crash plugin-container.exe with corrupt heap during local testing.

Crash 2
1. http://www.streamiz.com/film--streaming--page-13-lecteur-.html
2. sometimes crash with corrupt heap. This may be a loaded advertisement that is periodic. This may be related to bug 651575. I believe I did see an ad from ad.xtendmedia.com/ when testing this crash.

Automation found:

Operating system: Windows NT
                  5.1.2600 Service Pack 3
CPU: x86
     GenuineIntel family 6 model 44 stepping 2
     1 CPU

Crash reason:  EXCEPTION_ACCESS_VIOLATION_WRITE
Crash address: 0x38
Assertion: Unknown assertion type 0x00000000

Thread 0 (crashed)
 0  xul.dll!std::_Construct<MessageLoop::PendingTask,MessageLoop::PendingTask>(MessageLoop::PendingTask *,MessageLoop::PendingTask const &) [xmemory : 53 + 0x1
    eip = 0x01d4edeb   esp = 0x0012e738   ebp = 0x0012e744   ebx = 0x00000000
    esi = 0x00000001   edi = 0x07734988   eax = 0x00000038   ecx = 0x045641e0
    edx = 0x0012e7cc   efl = 0x00210202
    Found by: given as instruction pointer in context
 1  xul.dll!std::allocator<MessageLoop::PendingTask>::construct(MessageLoop::PendingTask *,MessageLoop::PendingTask const &) [xmemory : 156 + 0xc]
    eip = 0x01d4c814   esp = 0x0012e74c   ebp = 0x0012e758
    Found by: call frame info
 2  xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::push_back(MessageLoop::PendingTask const &) [deque : 826 + 0x2a]
    eip = 0x01d4bc46   esp = 0x0012e760   ebp = 0x0012e778
    Found by: call frame info
 3  xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::push(MessageLoop::PendingTask
    eip = 0x01d4ad73   esp = 0x0012e780   ebp = 0x0012e788
    Found by: call frame info
 4  xul.dll!MessageLoop::PostTask_Helper(tracked_objects::Location const &,Task *,int,bool) [message_loop.cc : 293 + 0x11]
    eip = 0x01d49f8f   esp = 0x0012e790   ebp = 0x0012e7e4
    Found by: call frame info
 5  xul.dll!MessageLoop::PostTask(tracked_objects::Location const &,Task *) [message_loop.cc : 251 + 0x13]
    eip = 0x01d49e1b   esp = 0x0012e7ec   ebp = 0x0012e800
    Found by: call frame info
 6  xul.dll!mozilla::ipc::RPCChannel::EnqueuePendingMessages() [RPCChannel.cpp : 366 + 0x5c]
    eip = 0x01b3512a   esp = 0x0012e808   ebp = 0x0012e844
    Found by: call frame info

original socorro report

bp-f3ef37b3-08e9-4a9a-a7e7-19e0b2110513 Crash Report [@ UserCallWinProcCheckWow ]

Comment 1

[smadayag]

tracking internally as #bug=2875277.  we are looking into it...

Comment 2

[smadayag]

was asked to close this based that Flash Player in not in the stack.

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: crash report 1 with NPSWF on stack

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: crash report 2 with NPSWF on stack

Comment 5

[Bob Clary [:bc] (inactive)]

I'm very disappointed in your early dismissal. I normally don't include full stacks, but will if necessary to get your attention.

Comment 6

[smadayag]

thanks. the full stack does help.

Comment 7

[smadayag]

bob - dev is not finding the 2 files as informative.  would you be able to attach the binary dump file?  thank you.

Comment 8

[Bob Clary [:bc] (inactive)]

I have the dmp file that Firefox produces and can attach that, but I doubt that is what they want. I bet they want the minidump created from windbg?

Comment 9

[Bob Clary [:bc] (inactive)]

I reproduced it twice with the myspace url by going to the page and attaching windbg to the plugin-container process then reloading. The saved dmp files are 131M and 158M so I'm not sure how you would like to handle it. You should be able to easily reproduce with a debug Firefox build and windbg.

In windbg it looks like:

HEAP[plugin-container.exe]: Invalid Address specified to RtlValidateHeap( 03360000, 03D8FC40 )
(11f4.d0c): Break instruction exception - code 80000003 (first chance)
eax=03d8fc38 ebx=03d8fc38 ecx=7c91d4fd edx=0012e2ce esi=03360000 edi=03360000
eip=7c90120e esp=0012e4d4 ebp=0012e4d8 iopl=0         nv up ei pl nz na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00200202
ntdll!DbgBreakPoint:
7c90120e cc              int     3

WARNING: Stack unwind information not available. Following frames may be wrong.
ntdll!DbgBreakPoint
ntdll!RtlpNtMakeTemporaryKey+0x6b72
ntdll!RtlValidateHeap+0xe0
kernel32!HeapValidate+0x14
MSVCR80D!CrtIsValidHeapPointer+0x15a
MSVCR80D!free_dbg+0x196
MSVCR80D!free_dbg+0x4e
MSVCR80D!free+0xe
mozalloc!moz_free(void * ptr = 0x03d8fc60)+0xd
xul!operator delete(void * ptr = 0x03d8fc60)+0xd
xul!std::allocator<std::_List_nod<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Node>::deallocate(struct std::_List_nod<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Node * _Ptr = 0x03d8fc60, unsigned int __formal = 1)+0x10
xul!std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::erase(class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1> _Where = class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1>)+0xe6
xul!stdext::_Hash<stdext::_Hmap_traits<int,mozilla::ipc::RPCChannel::RPCListener *,stdext::hash_compare<int,std::less<int> >,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> >,0> >::erase(class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1> _Where = class std::list<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *>,std::allocator<std::pair<int const ,mozilla::ipc::RPCChannel::RPCListener *> > >::_Iterator<1>)+0x8f
xul!IDMap<mozilla::ipc::RPCChannel::RPCListener>::Remove(int id = -275)+0x9c
xul!mozilla::plugins::PPluginModuleChild::Unregister(int aId = -275)+0x19
xul!mozilla::plugins::PPluginInstanceChild::Unregister(int aId = -275)+0x1e
xul!mozilla::plugins::PStreamNotifyChild::Unregister(int aId = -275)+0x1e
xul!mozilla::plugins::PStreamNotifyChild::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::RPCChannel::RPCListener>::ActorDestroyReason why = Deletion (1))+0x1f
xul!mozilla::plugins::PStreamNotifyChild::OnMessageReceived(class IPC::Message * __msg = 0x0012e8e0)+0x33e
xul!mozilla::plugins::PPluginModuleChild::OnMessageReceived(class IPC::Message * __msg = 0x0012e8e0)+0x62

It doesn't have flash on the stack however if that matters though I'm not sure how much to trust the stack especially since I don't know how to start plugin-container with the debugger already attached. I don't have a 3.5 debug build handy to try a non out of process plugin though.

Comment 10

[Bob Clary [:bc] (inactive)]

http://www.pspiso.com/ - Automation reproduced crash with Windows XP Nightly debug build and Flash 10.3.181.22 but not with any other combination of Windows XP, Windows 7 and 2.0, beta, aurora, nightly.

http://www.myspace.com/plies - Automation could not reproduce the crash with  at all.

Manually testing using the following steps on Windows XP:

1. Start MSVC
2. Start *DEBUG* build of Firefox
3. Navigate to cnn.com to activate plugin-container
4. Attach MSVC to plugin-container process
5. Load url
6. Shutdown

http://www.myspace.com/plies Nightly - Normal

http://www.myspace.com/plies Aurora  - breakpoint

 	ntdll.dll!7c90120e() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!7c96ee31() 	
 	ntdll.dll!7c96f26e() 	
 	ntdll.dll!7c962fe0() 	
 	ntdll.dll!7c90f65c() 	
>	msvcp80d.dll!_Mtxlock(_RTL_CRITICAL_SECTION * _Mtx=0x03340000)  Line 45	C
 	kernel32.dll!7c85f9a7() 	
 	msvcr80d.dll!_CrtIsValidHeapPointer(const void * pUserData=0x03d6dc90)  Line 2072	C++
 	msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x03d6dc90, int nBlockUse=1)  Line 1279 + 0x9 bytes	C++
 	msvcr80d.dll!_free_dbg(void * pUserData=0x03d6dc90, int nBlockUse=1)  Line 1220 + 0xd bytes	C++
 	msvcr80d.dll!free(void * pUserData=0x03d6dc90)  Line 1178 + 0xb bytes	C++
 	mozalloc.dll!moz_free(void * ptr=0x03d6dc90)  Line 94 + 0xa bytes	C++
 	xul.dll!operator delete(void * ptr=0x03d6dc90)  Line 253 + 0xa bytes	C++
 	xul.dll!std::allocator<MessageLoop::PendingTask>::deallocate(MessageLoop::PendingTask * _Ptr=0x03d6dc90, unsigned int __formal=1)  Line 141 + 0x9 bytes	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy()  Line 1254	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear()  Line 1081	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]())  Line 622 + 0x8 bytes	C++
 	xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]())  + 0x13 bytes	C++
 	xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]())  Line 19 + 0xc bytes	C++
 	xul.dll!MessageLoop::ReloadWorkQueue()  Line 385 + 0x16 bytes	C++
 	xul.dll!MessageLoop::DoWork()  Line 437	C++
 	xul.dll!base::MessagePumpForIO::DoRunLoop()  Line 454 + 0x17 bytes	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x03d2feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x03d2feb8)  Line 78 + 0x15 bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219	C++
 	xul.dll!MessageLoop::RunHandler()  Line 203	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!base::Thread::ThreadMain()  Line 159	C++
 	xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x0334eb38)  Line 27	C++
 	kernel32.dll!7c80b729() 	
 	xul.dll!mozilla::SVGLengthListSMILType::Add(nsSMILValue & aDest={...}, const nsSMILValue & aValueToAdd={...}, unsigned int aCount=2422607882)  Line 183 + 0x3e bytes	C++
 	xul.dll!nsDocShell::DoURILoad(nsIURI * aURI=0x296201dd, nsIURI * aReferrerURI=0x2b4555e4, int aSendReferrer=1298266322, nsISupports * aOwner=0xcf795128, const char * aTypeHint=0x904ee41d, nsIInputStream * aPostData=0xe021cd50, nsIInputStream * aHeadersData=0xa2b780c1, int aFirstParty=1654818883, nsIDocShell * * aDocShell=0xdde8de34, nsIRequest * * aRequest=0xd0e9a941, int aIsNewWindowTarget=1548465500, int aBypassClassifier=-1503536295, int aForceAllowCookies=-383164654)  Line 8813	C++

continuing hits invalid heap assertion

>	msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x03d6dc90, int nBlockUse=1)  Line 1279 + 0x30 bytes	C++
 	msvcr80d.dll!_free_dbg(void * pUserData=0x03d6dc90, int nBlockUse=1)  Line 1220 + 0xd bytes	C++
 	msvcr80d.dll!free(void * pUserData=0x03d6dc90)  Line 1178 + 0xb bytes	C++
 	mozalloc.dll!moz_free(void * ptr=0x03d6dc90)  Line 94 + 0xa bytes	C++
 	xul.dll!operator delete(void * ptr=0x03d6dc90)  Line 253 + 0xa bytes	C++
 	xul.dll!std::allocator<MessageLoop::PendingTask>::deallocate(MessageLoop::PendingTask * _Ptr=0x03d6dc90, unsigned int __formal=1)  Line 141 + 0x9 bytes	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy()  Line 1254	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear()  Line 1081	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]())  Line 622 + 0x8 bytes	C++
 	xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]())  + 0x13 bytes	C++
 	xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]())  Line 19 + 0xc bytes	C++
 	xul.dll!MessageLoop::ReloadWorkQueue()  Line 385 + 0x16 bytes	C++
 	xul.dll!MessageLoop::DoWork()  Line 437	C++
 	xul.dll!base::MessagePumpForIO::DoRunLoop()  Line 454 + 0x17 bytes	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x03d2feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x03d2feb8)  Line 78 + 0x15 bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 219	C++
 	xul.dll!MessageLoop::RunHandler()  Line 203	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!base::Thread::ThreadMain()  Line 159	C++
 	xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x0334eb38)  Line 27	C++
 	kernel32.dll!7c80b729() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	xul.dll!mozilla::SVGLengthListSMILType::Add(nsSMILValue & aDest={...}, const nsSMILValue & aValueToAdd={...}, unsigned int aCount=2422607882)  Line 183 + 0x3e bytes	C++
 	xul.dll!nsDocShell::DoURILoad(nsIURI * aURI=0x296201dd, nsIURI * aReferrerURI=0x2b4555e4, int aSendReferrer=1298266322, nsISupports * aOwner=0xcf795128, const char * aTypeHint=0x904ee41d, nsIInputStream * aPostData=0xe021cd50, nsIInputStream * aHeadersData=0xa2b780c1, int aFirstParty=1654818883, nsIDocShell * * aDocShell=0xdde8de34, nsIRequest * * aRequest=0xd0e9a941, int aIsNewWindowTarget=1548465500, int aBypassClassifier=-1503536295, int aForceAllowCookies=-383164654)  Line 8813	C++


http://www.myspace.com/plies Beta - Normal

http://www.myspace.com/plies 2.0.0 - breakpoint

 	ntdll.dll!7c90120e() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!7c96ee31() 	
 	ntdll.dll!7c96f26e() 	
 	ntdll.dll!7c962fe0() 	
 	ntdll.dll!7c91005d() 	
 	kernel32.dll!7c85f9a7() 	
>	msvcr80d.dll!_CrtIsValidHeapPointer(const void * pUserData=0x0419ace0)  Line 2072	C++
 	msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x0419ace0, int nBlockUse=1)  Line 1279 + 0x9 bytes	C++
 	msvcr80d.dll!_free_dbg(void * pUserData=0x0419ace0, int nBlockUse=1)  Line 1220 + 0xd bytes	C++
 	msvcr80d.dll!free(void * pUserData=0x0419ace0)  Line 1178 + 0xb bytes	C++
 	mozalloc.dll!moz_free(void * ptr=0x0419ace0)  Line 92 + 0xa bytes	C++
 	xul.dll!operator delete(void * ptr=0x0419ace0)  Line 253 + 0xa bytes	C++
 	xul.dll!std::allocator<MessageLoop::PendingTask *>::deallocate(MessageLoop::PendingTask * * _Ptr=0x0419ace0, unsigned int __formal=8)  Line 141 + 0x9 bytes	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy()  Line 1259	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear()  Line 1081	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]())  Line 622 + 0x8 bytes	C++
 	xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]())  + 0x13 bytes	C++
 	xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]())  Line 19 + 0xc bytes	C++
 	xul.dll!MessageLoop::ReloadWorkQueue()  Line 386 + 0x16 bytes	C++
 	xul.dll!MessageLoop::DoWork()  Line 438	C++
 	xul.dll!base::MessagePumpForIO::DoRunLoop()  Line 454 + 0x17 bytes	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x0418feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x0418feb8)  Line 78 + 0x15 bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 220	C++
 	xul.dll!MessageLoop::RunHandler()  Line 203	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!base::Thread::ThreadMain()  Line 159	C++
 	xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x039ce7d0)  Line 27	C++
 	kernel32.dll!7c80b729() 	
 	xul.dll!nsBindingManager::DropDocumentReference()  Line 1690 + 0xe bytes	C++

continuing hits invalid heap assertion

>	msvcr80d.dll!_free_dbg_nolock(void * pUserData=0x0419ace0, int nBlockUse=1)  Line 1279 + 0x30 bytes	C++
 	msvcr80d.dll!_free_dbg(void * pUserData=0x0419ace0, int nBlockUse=1)  Line 1220 + 0xd bytes	C++
 	msvcr80d.dll!free(void * pUserData=0x0419ace0)  Line 1178 + 0xb bytes	C++
 	mozalloc.dll!moz_free(void * ptr=0x0419ace0)  Line 92 + 0xa bytes	C++
 	xul.dll!operator delete(void * ptr=0x0419ace0)  Line 253 + 0xa bytes	C++
 	xul.dll!std::allocator<MessageLoop::PendingTask *>::deallocate(MessageLoop::PendingTask * * _Ptr=0x0419ace0, unsigned int __formal=8)  Line 141 + 0x9 bytes	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::_Tidy()  Line 1259	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::clear()  Line 1081	C++
 	xul.dll!std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> >::operator=(const std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > & _Right=[0]())  Line 622 + 0x8 bytes	C++
 	xul.dll!std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > >::operator=(const std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & __that=[0]())  + 0x13 bytes	C++
 	xul.dll!std::swap<std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > >(std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Left=[0](), std::queue<MessageLoop::PendingTask,std::deque<MessageLoop::PendingTask,std::allocator<MessageLoop::PendingTask> > > & _Right=[0]())  Line 19 + 0xc bytes	C++
 	xul.dll!MessageLoop::ReloadWorkQueue()  Line 386 + 0x16 bytes	C++
 	xul.dll!MessageLoop::DoWork()  Line 438	C++
 	xul.dll!base::MessagePumpForIO::DoRunLoop()  Line 454 + 0x17 bytes	C++
 	xul.dll!base::MessagePumpWin::RunWithDispatcher(base::MessagePump::Delegate * delegate=0x0418feb8, base::MessagePumpWin::Dispatcher * dispatcher=0x00000000)  Line 54	C++
 	xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate * delegate=0x0418feb8)  Line 78 + 0x15 bytes	C++
 	xul.dll!MessageLoop::RunInternal()  Line 220	C++
 	xul.dll!MessageLoop::RunHandler()  Line 203	C++
 	xul.dll!MessageLoop::Run()  Line 177	C++
 	xul.dll!base::Thread::ThreadMain()  Line 159	C++
 	xul.dll!`anonymous namespace'::ThreadFunc(void * closure=0x039ce7d0)  Line 27	C++
 	kernel32.dll!7c80b729() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	xul.dll!nsBindingManager::DropDocumentReference()  Line 1690 + 0xe bytes	C++

http://www.pspiso.com/ Nightly - Normal

http://www.pspiso.com/ Aurora - Normal

http://www.pspiso.com/ Beta - Normal

http://www.pspiso.com/ 2.0.0 - Normal

Comment 11

[Bob Clary [:bc] (inactive)]

Sitting on http://www.pspiso.com/ in Aurora I hit

// In non-debug builds, this method is declared as an empty inline method.
#ifndef NDEBUG
void LockImpl::AssertAcquired() const {
  DCHECK(recursion_count_shadow_ > 0);
=>  DCHECK_EQ(owning_thread_id_, PlatformThread::CurrentId());
}

 	NPSWF32.dll!0429fc8f() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for NPSWF32.dll]	
 	NPSWF32.dll!04526a8e() 	
 	NPSWF32.dll!0415b2f3() 	
 	NPSWF32.dll!0422f70f() 	
 	NPSWF32.dll!0421b2e2() 	
 	NPSWF32.dll!0422f8e3() 	
 	NPSWF32.dll!0422ff33() 	
 	NPSWF32.dll!043187ff() 	
 	NPSWF32.dll!0423041d() 	
 	NPSWF32.dll!042c1f28() 	
 	NPSWF32.dll!0423060c() 	
 	NPSWF32.dll!04230617() 	
 	NPSWF32.dll!042305b1() 	
 	NPSWF32.dll!042a185a() 	
>	xul.dll!LockImpl::AssertAcquired()  Line 71 + 0x5 bytes	C++
 	xul.dll!Lock::Release()  Line 17 + 0xf bytes	C++
 	cdcdcd00()	

Looks like uninitialized memory got called. Sorry but I don't have Flash symbols for 22 handy. I tried to reproduce by loading the page and letting it sit idle for a while and by reloading multiple times but could not.

Comment 12

[Benjamin Smedberg]

This appears to be heap corruption. Probably none of the stacks are going to point to the actual problem: we need to find the point of corruption, which may only be possible using a valgrind-like tool or VMWare recording.

Comment 13

[Bob Clary [:bc] (inactive)]

This appears to have been fixed in Flash 10.3.181.26 as I can not longer reproduce. Leaving confidential for now.

Comment 14

[Bob Clary [:bc] (inactive)]

With Flash 10.3.183.5

http://www.blogdelnarco.com/2011/07/destazan-brutalmente-hombre-en-morelos.html#more

Locally on Windows XP I see corrupted heap. In automation I saw the original signature. 

http://www.ceropeliculasmegavideo.net/search?updated-max=2011-07-17T18%25253A14%25253A01-04%25253A30%2526max-results=42%2526popening=2

On reload I hit bug 641226
xul.dll!mozilla::plugins::PPluginInstanceChild::FatalError(const char * const msg=0x026d70e4)  Line 2367 + 0x18 bytes	C++

Comment 15

[smadayag]

we re-opened our internal bug, #2875277.  it is currently under review.  Reproduced an Access violation write on a XP VM with 10,3,183,5...

Comment 16

[Bob Clary [:bc] (inactive)]

Jeff,

blogdelnarco.com appears to have been taken down due to threats against bloggers in mexico.

I retested all of the crashes I have seen with Flash 10.3.183.10. I'll detail the ones that result in debug crt assetions here. I just confirmed these on Windows XP and Firefox 7 from the beta channel.

http://www.incontri-x-sesso.com/?id=1%25252525252526track=IT-Exit-Megasesso
NSFW (mild). You may need to reload and/or mouse over the flash images to see the problem.

http://www.outofaces.com/lucy-pinder/
NSFW (mild)

http://www.plunderguide.com/aryane-steinkopf/aryane-steinkopf-4/
NSFW (mild)

bug 587982 contains patches to force aborts when MS CRT debug assertions fire. It is helpful but not necessary. Debug builds of Firefox are necessary to show the debug MS CRT assertions and errors however.

Comment 17

[Bob Clary [:bc] (inactive)]

Attachment: 2011-11-26-results.html

Darin,

Today, I retested the urls in this bug on Windows XP and Windows 7 with Flash 11.1.102.55 using debug builds of Firefox which include the patch from bug 587982 on Beta/9, Aurora/10, Nightly/11 and reproduced one ABORT: CRT ASSERT dbgheap.c(1279) : Assertion failed: _CrtIsValidHeapPointer(pUserData) on Aurora WIndows XP with signature 

mozalloc_abort(char const* const) | MSCRTReportHook moz_free operator delete(void*) mozilla::plugins::StreamNotifyChild::`vector deleting destructor'(unsigned int) + 0x1f mozilla::plugins::BrowserStreamChild::Deliver() 	NULL

http://www.incontri-x-sesso.com/?id=1%25252525252526track=IT-Exit-Megasesso (NSFW)

I have attached the full set of reproducible Flash crashes from the 11/26 Flash crash retest. Additional crashes and debug heap assertions have been seen since then, but this should be a good start. Note that some of these urls are extremely NSFW.

Please use a debug build of Firefox on Windows XP with the appropriate patch from bug 587982. If you have any questions please contact me.

Thanks for looking into this.

Comment 18

[Bob Clary [:bc] (inactive)]

Attachment: testcase (sort of)

Start Visual Studio or your debugger of choice, then open this file in a debug build of Firefox. Once the page loads, attach the debugger to the plugin-container.exe process and wait for it to hit the CRT assertion. It may take some time, but it should hit the CRT assertion. I was able to hit the assertion fairly regularly up until mid-morning. I think you will need to leave it open until the ad rotation brings back the offending content.

This loads SFW advertisements.

Sorry, I haven't been able to get the actual flash file that asserts.

Comment 19

[Bob Clary [:bc] (inactive)]

Of course, after being reproducible all last night and this morning I have been running the test page all day and not hit the assert once. I'll keep that one running but will start looking for another example....

Comment 20

[Bob Clary [:bc] (inactive)]

Attachment: another testcase

This reproduces for me. It appears the number of simultaneous ads helps.

Comment 21

[Bob Clary [:bc] (inactive)]

A related assertion that may be of interest is:

###!!! ASSERTION: nsChannelClassifier not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/netwerk/base/src/nsChannelClassifier.cpp, line 57

Comment 22

[Bob Clary [:bc] (inactive)]

as well as

###!!! ASSERTION: mozHunspellDirProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/extensions/spellcheck/hunspell/src/mozHunspellDirProvider.cpp, line 45
###!!! ASSERTION: mozHunspellDirProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/extensions/spellcheck/hunspell/src/mozHunspellDirProvider.cpp, line 45
###!!! ASSERTION: DirectoryProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/browser/components/dirprovider/DirectoryProvider.cpp, line 65
###!!! ASSERTION: DirectoryProvider not thread-safe: '_mOwningThread.GetThread() == PR_GetCurrentThread()', file c:/work/mozilla/builds/nightly/mozilla/browser/components/dirprovider/DirectoryProvider.cpp, line 65

Comment 23

[Bob Clary [:bc] (inactive)]

Johnny, can you look at this? I can reliably reproduce this on Windows XP but Adobe is having problems. The stacks I see don't have Flash on the stack though that is not a reliable indicator that is it not Flash, but it could just as easily be our fault. 

Start debug build of Firefox trunk
Go to adobe.com/flashplayer to instantiate the plugin-container process.
Attach Visual Studio's debugger to plugin-container.exe
Open both test case attachments in bug 657588 in separate Windows.
Assert (time may vary depending on ad rotations)

Comment 24

[Benjamin Smedberg]

I think the only way we're going to make progress on this is with either record-replay or a purify build, since this involves heap corruption which happens before the crash. I don't have access to either of those tools right now. Whoever does those runs will need Flash PDB files.

Comment 25

[Bob Clary [:bc] (inactive)]

We have purify on sm-purify01 but I find it exceedingly difficult to get it to do anything useful with Firefox. There is a record-replay vm set up in the office right?

Comment 26

[Johnny Stenback (:jst)]

Yes, we have a record and replay box set up in the office. No flash symbols, but a good first step would be to find out whether it's us or flash that's corrupting memory here, if it's flash, we can work on getting access to the necessary symbols on the replay box.

Kyle has had good success with the replay tools before. Kyle, can you take a stab at this? If not, please re-assign back to smadayag until we find a better owner. If we could get a stack that shows where the corruption is coming from then we can find the right person to work on that, whether it's someone at Mozilla or someone at Adobe.

Bob, is this only happening in WinXP? I believe we have Win7 installed in the replay VM, but I also believe it's possible to record and do replay debugging in a WinXP VM. IIRC we even used to have one set up, but I'm not sure if we do at this point.

Comment 27

[Bob Clary [:bc] (inactive)]

Yes, this is XP only.

Comment 28

[Bob Clary [:bc] (inactive)]

I've been reading up on this assertion and it appears one of the cases where this may occur is when memory is allocated in one dll and then passed to another dll where each dll uses a different crt. Especially problematic is the case where a dll frees heap allocated in a different dll.

http://msdn.microsoft.com/en-us/library/ms235460%28v=vs.80%29.aspx

I assume the flash dll is using a different crt than our debug build and wonder if that may be the cause of the CRT Assert? Even in a release build we would be using different crt I think since a release build of Firefox will be using a patched version of the crt that includes jemalloc support while Flash uses something different I presume.

Comment 29

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Bob Clary [:bc:] from comment #28)
> I assume the flash dll is using a different crt than our debug build and
> wonder if that may be the cause of the CRT Assert? Even in a release build
> we would be using different crt I think since a release build of Firefox
> will be using a patched version of the crt that includes jemalloc support
> while Flash uses something different I presume.

I'm hardly an NPAPI expert but this is what NPN_MemAlloc is for, right?

dumpbin.exe -IMPORTS NPSWF32.dll shows no load time dependencies on any CRT.  I think this is more likely to be actual memory corruption than a heap mismatch.

Comment 30

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I've got an XP VM set up for record and replay.  Attempting to reproduce.

Comment 31

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

No luck with the testcase, and bc had no luck with the other URLs he tried manually.  He's going to have the automation retest the URLs and we'll go from there.

Comment 32

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

bc found a URL that asserted repeatedly for him, and I was able to capture the assertion in the replay setup.

Comment 33

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Unfortunately I haven't made much progress determining what exactly is going wrong in the recording.  I have some other stuff I need to finish up by the end of the quarter so I'm not going to get back to this for a week or two.

Comment 35

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

This is a bug in Gecko.  More details and a patch in a few minutes.

Comment 36

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: Patch

I managed to get this in a replay machine and isolate the code causing the memory corruption.  After sitting down with bent and getting some lessons on how all this IPC plugins stuff works, we determined that the problem is the following:

At http://hg.mozilla.org/mozilla-central/annotate/e0d9c8ddd5bd/dom/plugins/ipc/PluginModuleChild.cpp#l1067 we create a StreamNotifyChild object and pass it to CallPStreamNotifyConstructor.

We end up at http://hg.mozilla.org/mozilla-central/annotate/e0d9c8ddd5bd/dom/plugins/ipc/PluginInstanceParent.cpp#l440 in the parent process.  This function ensures that if the child was not told to delete its stream notify actor and the constructor failed, that the child is told to delete its stream notify actor.  It does not, however, ensure that if the child is told to delete its stream notify actor that an error code is returned.  Thus we return from the ctor with NPERR_ERROR_OK but the stream notify actor is already gone.

We return to the child side, where we see

1085 if (NPERR_NO_ERROR == err) {
1086     // If NPN_PostURLNotify fails, the parent will immediately send us
1087     // a PStreamNotifyDestructor, which should not call NPP_URLNotify.
1088     sn->SetValid(aNotifyData); 
1089 }

Since err is NPERR_NO_ERR we write to the freed memory and cause the heap corruption that later causes the assertions/crashes.

This patch ensures that if the stream notify actor was destroyed that we return an error code (NPERR_GENERIC_ERR) and thus avoid the heap corruption.

Comment 37

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

bc is going to rerun automation with this patch and see if he still sees any of these crashes/asserts.

Comment 38

[Benjamin Smedberg]

Comment on attachment 595229 [details] [diff] [review]
Patch

This does not appear to be the patch you meant to attach.

Comment 39

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Attachment: Patch

And I attached the wrong file.

Comment 40

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/mozilla-central/rev/b584b4be0a00

Comment 41

[Bob Clary [:bc] (inactive)]

This appears to have helped. Reproducing the heap corruption on nightly has been difficult so the fact I did not see it after this patch is not a guarantee. But this also eliminated ( I think ) the related FatalError crash on Nightly in bug 641226. It would really really help to make sure if we can get this on both aurora and beta asap.

Comment 42

[Alex Keybl [:akeybl]]

From my read of the patch this looks low risk, but looks can of course be deceiving. Would you all mind providing a risk evaluation?

Comment 43

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Plugin code is kind of hairy, but this is pretty low risk.  I suppose there is potential for this to cause us to return an error to plugins where they weren't getting it before, but at the moment instead of getting that error they're just crashing (or worse!).

Comment 44

[Benjamin Smedberg]

I think this is about as safe as a plugin bug can be.

Comment 45

[Alex Keybl [:akeybl]]

Comment on attachment 595432 [details] [diff] [review]
Patch

[Triage Comment]
Given the low-risk evaluation, approving for Aurora 12 and Beta 11.

Comment 46

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/46a2e9a276f1
https://hg.mozilla.org/releases/mozilla-beta/rev/cea20ca3ed3c

Comment 47

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We also want this on 10esr and probably on 1.9.2 as well (the code is the same there).

Comment 48

[Bob Clary [:bc] (inactive)]

I rebuilt beta and aurora and retested with urls from the last week and was not able to reproduce the abort or the related crashes. Thanks Kyle, you rock!

Comment 51

[Alex Keybl [:akeybl]]

Please land on mozilla-esr10 as soon as possible. For more information on the landing process, please see https://wiki.mozilla.org/Release_Management/ESR_Landing_Process

Comment 52

[Lukas Blakk [:lsblakk] use ?needinfo]

[Triage comment]

This bug is being tracked for landing on ESR branch.  Please land patches on http://hg.mozilla.org/releases/mozilla-esr10/by Thursday March 1, 2012 in order to be ready for go-to-build on Friday March 2, 2012.

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more information.

Comment 53

[Daniel Veditz [:dveditz]]

Comment on attachment 595432 [details] [diff] [review]
Patch

Approved for 1.9.2.28, a=dveditz for release-drivers

Comment 54

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

https://hg.mozilla.org/releases/mozilla-esr10/rev/83533cc02c57
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/9b893007fa4e

Comment 55

[Al Billings [:abillings - ex-MoCo]]

(In reply to Bob Clary [:bc:] from comment #48)
> I rebuilt beta and aurora and retested with urls from the last week and was
> not able to reproduce the abort or the related crashes. Thanks Kyle, you
> rock!

Was this with Firefox 11 and 12 then? I'm trying to figure out if we need to verify it in those branches or if you already did so, Bob? Flags for those releases are still marked as "fixed" and not "verfied" so I am asking.

Comment 56

[u279076]

I'm unable to reproduce the crash using Flash 11.1.102.63 so I can't verify if this is fixed for Firefox 10.0.3esr. Does this require a particular version of Flash?

Comment 57

[juan becerra [:juanb]]

I've been trying to reproduce the issue using affected builds and Flash version 10.3.181.14, but I haven't been able to. I don't have a build environment handy to check with the more detailed instructions.

Bob, could you help us verify this on your environment?

Comment 58

[Bob Clary [:bc] (inactive)]

This wasn't a problem in flash but in our code and only appeared with debug builds on windows xp. I've already verified this for Beta/11, Aurora/12, and Nightly/13. I haven't seen the ms crt debug asserts in some time. I don't have a 1.9.2 or 10 environment available but the patch is so simple that I don't expect them to be any different.

Juan, I assume 10.3.181.14 means you were testing with Mac OS X 10.5?

Comment 59

[juan becerra [:juanb]]

I was testing on XP. I got an older version of Flash (the one mentioned in comment #0) from Adobe's archives.

Comment 60

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Have you been trying to verify this with debug builds?

Comment 61

[juan becerra [:juanb]]

No, I was feeling lucky thinking I could reproduce the crash without having to go use debug builds, and that's why I asked for bc's help. I'll report back when after trying debug builds.

Comment 62

[u279076]

Tried the esr10 debug build from February 22nd, 2012 with Flash 10.3.181.14 on Windows XP and I can't reproduce the original crash -- therefor, I am unable to verify the fix.

Comment 63

[Bob Clary [:bc] (inactive)]

I was able to reproduce it with ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-02-25-mozilla-esr10-debug/ and NSFW http://www.incontri-x-sesso.com/?id=1%25252525252526track=IT-Exit-Megasesso 

I was not able to reproduce with ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-esr10/firefox-10.0.4esrpre.en-US.win32.zip

Anthony + Juan: You can do which ever qa stamps you need for esr.

Comment 64

[u279076]

Thanks Bob, based on your testing I'm going to call this verified for ESR. If you have a chance could you quickly verify for Firefox 3.6.28 as well?

Comment 65

[Bob Clary [:bc] (inactive)]

I couldn't reproduce the debug assertion or crash with 1.9.2. Btw in comment 63 I incorrectly verified with a non debug build. I retested with a debug build and verified 10esr did not assert/crash.

Comment 66

[u279076]

(In reply to Bob Clary [:bc:] from comment #65)
> I couldn't reproduce the debug assertion or crash with 1.9.2. 

With a 1.9.2 build from before or after this was fixed in 3.6.28?

Comment 67

[Bob Clary [:bc] (inactive)]

either

Comment 68

[u279076]

(In reply to Bob Clary [:bc:] from comment #67)
> either

Is there anything we can do to verify this fixed for Firefox 3.6.28 then? I'm guessing no since it's not reproducible in the first place.

Comment 69

[Bob Clary [:bc] (inactive)]

Not easily that I can think of. I could try to manually test the list of urls where I have seen this before but that is a sink of time and not worth the effort imo.

Comment 70

[u279076]

(In reply to Bob Clary [:bc:] from comment #69)
> Not easily that I can think of. I could try to manually test the list of
> urls where I have seen this before but that is a sink of time and not worth
> the effort imo.

Given that, QA won't spend any more time on this either. It's been verified everywhere else, so I have a fair bit of confidence in the assumption that this is okay for 3.6.28.

Comment 71

[watch party]

Netflix Watch Party is an extension that allows you to create or join the watch parties with your long-distance friends. It enables you to add as many members as you want to your Netflix party and watch Netflix together.
.
..
http://netflixwatchparty.net