1319271 - IDB - Use After Free in ipc::IPCResult::Fail

Status: RESOLVED FIXED

(Core :: IPC, defect, P1)

Description

[Looben Yang]

Reproduction test case (just the worker code, full server code in UAF_Fail_Repro.js ):

	workercode1 += 'var  db;\n';
	workercode1 += 'var dbreq0 ;  dbreq0= indexedDB.open("TestDb'+dbSuffix+'",  1);\n';
	workercode1 += ' dbreq0.onupgradeneeded = function(event) { db = event.target.result;console.log("------upgrade needed.");db.onversionchange=function(event) {try{ dbreq0 = indexedDB.open("TestDb'+dbSuffix+'",  3);}catch(e){}\n';
	workercode1 += '};\n';
	workercode1 += '};\n';


Steps to reproduce: 
	1. Run server side script UAF_Fail_Repro.js in Node.js (node UAF_Fail_Repro.js).
	2. Enter http://localhost:12345 in Firefox browser.


Firefox version: 53.0a1 (2016-11-21) (32-bit)
OS: Windows 10


Stack trace:

	(5bac.4fb4): Access violation - code c0000005 (!!! second chance !!!)
	eax=e5e5e5e5 ebx=23239dc0 ecx=0e1c2290 edx=1163f41c esi=14a3f82f edi=23239dc0
	eip=1007aeb2 esp=14a3f7c8 ebp=14a3f7fc iopl=0         nv up ei pl nz na pe nc
	cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
	xul!mozilla::ipc::IPCResult::Fail+0x1e:
	1007aeb2 ff5028          call    dword ptr [eax+28h]  ds:002b:e5e5e60d=????????
	0:055> .exr -1
	ExceptionAddress: 1007aeb2 (xul!mozilla::ipc::IPCResult::Fail+0x0000001e)
	   ExceptionCode: c0000005 (Access violation)
	  ExceptionFlags: 00000000
	NumberParameters: 2
	   Parameter[0]: 00000000
	   Parameter[1]: e5e5e60d
	Attempt to read from address e5e5e60d
	0:055> kb
	 # ChildEBP RetAddr  Args to Child              
	00 14a3f7fc 109a195f 0e1c2290 146aa000 0e1c2290 xul!mozilla::ipc::IPCResult::Fail+0x1e [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\glue\protocolutils.cpp @ 61]
	01 14a3f814 0fe74ffa 14a3f82f 2311ac78 2311ac00 xul!mozilla::dom::indexedDB::`anonymous namespace'::VersionChangeTransaction::RecvDeleteMe+0x3b [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\dom\indexeddb\actorsparent.cpp @ 15931]
	02 14a3f8e4 0fac45c2 23239dc0 2311ac78 00000001 xul!mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent::OnMessageReceived+0x3b074e [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\obj-firefox\ipc\ipdl\pbackgroundidbversionchangetransactionparent.cpp @ 180]
	03 14a3fa80 0f82fed1 23239dc0 2311ac78 2beffd70 xul!mozilla::ipc::PBackgroundParent::OnMessageReceived+0x48 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\obj-firefox\ipc\ipdl\pbackgroundparent.cpp @ 1897]
	04 14a3fa9c 0f82fd3f 00239dc0 23239dc0 2311ac78 xul!mozilla::ipc::MessageChannel::DispatchAsyncMessage+0x50 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\glue\messagechannel.cpp @ 1744]
	05 14a3fb04 0f82f409 23239dc0 23239da0 2beffd70 xul!mozilla::ipc::MessageChannel::DispatchMessageW+0xb7 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\glue\messagechannel.cpp @ 1684]
	06 14a3fb1c 0f82f3a4 23239da0 00000000 23239da0 xul!mozilla::ipc::MessageChannel::RunMessage+0x53 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\glue\messagechannel.cpp @ 1573]
	07 14a3fb34 0f82b66f 23239da0 14611600 00000001 xul!mozilla::ipc::MessageChannel::MessageTask::Run+0x38 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\glue\messagechannel.cpp @ 1598]
	08 14a3fbcc 0f82a0b0 145d4b60 00000001 14a3fbe7 xul!nsThread::ProcessNextEvent+0x2aa [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\xpcom\threads\nsthread.cpp @ 1219]
	09 14a3fbe8 0f8dc250 14611600 14611600 581b9a8d xul!NS_ProcessNextEvent+0x16 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\xpcom\glue\nsthreadutils.cpp @ 361]
	0a 14a3fc08 0f8dc148 00611600 0eff6cb5 14611600 xul!mozilla::ipc::MessagePumpForNonMainThreads::Run+0xc0 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\glue\messagepump.cpp @ 369]
	0b (Inline) -------- -------- -------- -------- xul!MessageLoop::RunInternal+0x8 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\chromium\src\base\message_loop.cc @ 232]
	0c 14a3fc40 0f8dc117 145d4b60 00000001 581b9a00 xul!MessageLoop::RunHandler+0x20 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\chromium\src\base\message_loop.cc @ 226]
	0d 14a3fc60 0f8db01d 770b29d0 14439a30 14439ae0 xul!MessageLoop::Run+0x19 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\ipc\chromium\src\base\message_loop.cc @ 206]
	0e 14a3fc94 581b9ff2 145d4b60 06d421c8 581afab5 xul!nsThread::ThreadFunc+0xb8 [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\xpcom\threads\nsthread.cpp @ 469]
	0f 14a3fcb4 581afac2 14439a30 14a3fcfc 6c8962a4 nss3!_PR_NativeRunThread+0xcc [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\nsprpub\pr\src\threads\combined\pruthr.c @ 397]
	10 14a3fcc0 6c8962a4 14439a30 7648327f 6c896250 nss3!pr_root+0xd [c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\src\nsprpub\pr\src\md\windows\w95thred.c @ 95]
	11 14a3fcfc 770b38f4 06d421c8 770b38d0 6dd351a8 ucrtbase!_crt_at_quick_exit+0x104
	12 14a3fd10 77bf5de3 06d421c8 6d0faeba 00000000 KERNEL32!BaseThreadInitThunk+0x24
	WARNING: Stack unwind information not available. Following frames may be wrong.
	13 14a3fd58 77bf5dae ffffffff 77c1b7bd 00000000 ntdll!RtlUnicodeStringToInteger+0x253
	14 14a3fd68 00000000 6c896250 06d421c8 00000000 ntdll!RtlUnicodeStringToInteger+0x21e



Variables shows that the actor object had been freed:

	actor	class mozilla::NotNull<mozilla::ipc::IProtocol *>
	 mBasePtr	0x0e1c2290 class mozilla::ipc::IProtocol *
	  mozilla::ipc::HasResultCodes	struct mozilla::ipc::HasResultCodes
	  __vfptr	0xe5e5e5e5 
	  mId	0n-437918235
	  mSide	0n-437918235 (No matching enumerant)
	  mManager	0xe5e5e5e5 class mozilla::ipc::IProtocol *
	  mChannel	0xe5e5e5e5 class mozilla::ipc::MessageChannel *
	where	0x1163f41c "RecvDeleteMe"
	errorMsg	class nsPrintfCString

Comment 1

[Looben Yang]

Attachment: UAF_Fail_Repro.js

Comment 2

[Andrew Overholt [:overholt]]

Jan or Bevis, can one of you take a look ASAP? Thanks.

Comment 3

[Looben Yang]

Ran the same test case with longer refresh timer (600ms -> 1000ms) in a customized Linux ASAN build with thread safety assert disabled, the ASAN build did report a Use After Free:


Firefox version: 53.0a1 (2016-11-21) (64-bit)

=================================================================
==5720==ERROR: AddressSanitizer: heap-use-after-free on address 0x611001b4c8e0 at pc 0x7f11bd869cf7 bp 0x7f119b071b30 sp 0x7f119b071b28
READ of size 8 at 0x611001b4c8e0 thread T29 (IPDL Background)
    #0 0x7f11bd869cf6 in mozilla::ipc::IPCResult::Fail(mozilla::NotNull<mozilla::ipc::IProtocol*>, char const*, char const*) /home/coder/OpenSrcCode/firefox/ipc/glue/ProtocolUtils.cpp:61:43
    #1 0x7f11bdfdcf4f in mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent::OnMessageReceived(IPC::Message const&) /home/coder/OpenSrcCode/firefox/objdir-ff-asan/ipc/ipdl/PBackgroundIDBVersionChangeTransactionParent.cpp:180:20
    #2 0x7f11be00bbf2 in mozilla::ipc::PBackgroundParent::OnMessageReceived(IPC::Message const&) /home/coder/OpenSrcCode/firefox/objdir-ff-asan/ipc/ipdl/PBackgroundParent.cpp:817:16
    #3 0x7f11bd84e8a9 in mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&) /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1743:14
    #4 0x7f11bd84b2a6 in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1681:17
    #5 0x7f11bd84d631 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1572:5
    #6 0x7f11bd84dc84 in mozilla::ipc::MessageChannel::MessageTask::Run() /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1597:5
    #7 0x7f11bc7da84b in nsThread::ProcessNextEvent(bool, bool*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:1213:7
    #8 0x7f11bc866888 in NS_ProcessNextEvent(nsIThread*, bool) /home/coder/OpenSrcCode/firefox/xpcom/glue/nsThreadUtils.cpp:361:10
    #9 0x7f11bd856d26 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/coder/OpenSrcCode/firefox/ipc/glue/MessagePump.cpp:368:5
    #10 0x7f11bd748188 in RunInternal /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:232:3
    #11 0x7f11bd748188 in RunHandler /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:225
    #12 0x7f11bd748188 in MessageLoop::Run() /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:205
    #13 0x7f11bc7d4b5b in nsThread::ThreadFunc(void*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:467:5
    #14 0x7f11d6ae8b96 in _pt_root /home/coder/OpenSrcCode/firefox/nsprpub/pr/src/pthreads/ptthread.c:216:5
    #15 0x7f11d6737181 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8181)
    #16 0x7f11d582947c in clone (/lib/x86_64-linux-gnu/libc.so.6+0xfa47c)

0x611001b4c8e0 is located 96 bytes inside of 240-byte region [0x611001b4c880,0x611001b4c970)
freed by thread T29 (IPDL Background) here:
    #0 0x4c4e90 in __interceptor_free /home/coder/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:38
    #1 0x7f11c2ad8900 in Release /home/coder/OpenSrcCode/firefox/dom/indexedDB/ActorsParent.cpp:6784:3
    #2 0x7f11c2ad8900 in Release /home/coder/OpenSrcCode/firefox/objdir-ff-asan/dist/include/mozilla/RefPtr.h:40
    #3 0x7f11c2ad8900 in Release /home/coder/OpenSrcCode/firefox/objdir-ff-asan/dist/include/mozilla/RefPtr.h:399
    #4 0x7f11c2ad8900 in ~RefPtr /home/coder/OpenSrcCode/firefox/objdir-ff-asan/dist/include/mozilla/RefPtr.h:78
    #5 0x7f11c2ad8900 in mozilla::dom::indexedDB::(anonymous namespace)::Database::DeallocPBackgroundIDBVersionChangeTransactionParent(mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent*) /home/coder/OpenSrcCode/firefox/dom/indexedDB/ActorsParent.cpp:14470
    #6 0x7f11bdfd944d in mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent::Send__delete__(mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent*) /home/coder/OpenSrcCode/firefox/objdir-ff-asan/ipc/ipdl/PBackgroundIDBVersionChangeTransactionParent.cpp:107:5
    #7 0x7f11c2b3d8cd in RecvDeleteMe /home/coder/OpenSrcCode/firefox/dom/indexedDB/ActorsParent.cpp:15930:8
    #8 0x7f11c2b3d8cd in non-virtual thunk to mozilla::dom::indexedDB::(anonymous namespace)::VersionChangeTransaction::RecvDeleteMe() /home/coder/OpenSrcCode/firefox/dom/indexedDB/ActorsParent.cpp:7114
    #9 0x7f11bdfdcf4f in mozilla::dom::indexedDB::PBackgroundIDBVersionChangeTransactionParent::OnMessageReceived(IPC::Message const&) /home/coder/OpenSrcCode/firefox/objdir-ff-asan/ipc/ipdl/PBackgroundIDBVersionChangeTransactionParent.cpp:180:20
    #10 0x7f11be00bbf2 in mozilla::ipc::PBackgroundParent::OnMessageReceived(IPC::Message const&) /home/coder/OpenSrcCode/firefox/objdir-ff-asan/ipc/ipdl/PBackgroundParent.cpp:817:16
    #11 0x7f11bd84e8a9 in mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&) /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1743:14
    #12 0x7f11bd84b2a6 in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1681:17
    #13 0x7f11bd84d631 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1572:5
    #14 0x7f11bd84dc84 in mozilla::ipc::MessageChannel::MessageTask::Run() /home/coder/OpenSrcCode/firefox/ipc/glue/MessageChannel.cpp:1597:5
    #15 0x7f11bc7da84b in nsThread::ProcessNextEvent(bool, bool*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:1213:7
    #16 0x7f11bc866888 in NS_ProcessNextEvent(nsIThread*, bool) /home/coder/OpenSrcCode/firefox/xpcom/glue/nsThreadUtils.cpp:361:10
    #17 0x7f11bd856d26 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/coder/OpenSrcCode/firefox/ipc/glue/MessagePump.cpp:368:5
    #18 0x7f11bd748188 in RunInternal /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:232:3
    #19 0x7f11bd748188 in RunHandler /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:225
    #20 0x7f11bd748188 in MessageLoop::Run() /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:205
    #21 0x7f11bc7d4b5b in nsThread::ThreadFunc(void*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:467:5
    #22 0x7f11d6ae8b96 in _pt_root /home/coder/OpenSrcCode/firefox/nsprpub/pr/src/pthreads/ptthread.c:216:5
    #23 0x7f11d6737181 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8181)

previously allocated by thread T29 (IPDL Background) here:
    #0 0x4c5198 in __interceptor_malloc /home/coder/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:52
    #1 0x4fd6ad in moz_xmalloc /home/coder/OpenSrcCode/firefox/memory/mozalloc/mozalloc.cpp:83:17
    #2 0x7f11c2aba92f in operator new /home/coder/OpenSrcCode/firefox/objdir-ff-asan/dist/include/mozilla/mozalloc.h:194:12
    #3 0x7f11c2aba92f in mozilla::dom::indexedDB::(anonymous namespace)::OpenDatabaseOp::BeginVersionChange() /home/coder/OpenSrcCode/firefox/dom/indexedDB/ActorsParent.cpp:21888
    #4 0x7f11c2ab13c9 in mozilla::dom::indexedDB::(anonymous namespace)::FactoryOp::Run() /home/coder/OpenSrcCode/firefox/dom/indexedDB/ActorsParent.cpp:21082:12
    #5 0x7f11bc7da84b in nsThread::ProcessNextEvent(bool, bool*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:1213:7
    #6 0x7f11bc866888 in NS_ProcessNextEvent(nsIThread*, bool) /home/coder/OpenSrcCode/firefox/xpcom/glue/nsThreadUtils.cpp:361:10
    #7 0x7f11bd856d26 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /home/coder/OpenSrcCode/firefox/ipc/glue/MessagePump.cpp:368:5
    #8 0x7f11bd748188 in RunInternal /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:232:3
    #9 0x7f11bd748188 in RunHandler /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:225
    #10 0x7f11bd748188 in MessageLoop::Run() /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:205
    #11 0x7f11bc7d4b5b in nsThread::ThreadFunc(void*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:467:5
    #12 0x7f11d6ae8b96 in _pt_root /home/coder/OpenSrcCode/firefox/nsprpub/pr/src/pthreads/ptthread.c:216:5
    #13 0x7f11d6737181 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8181)

Thread T29 (IPDL Background) created by T0 here:
    #0 0x42f3b9 in __interceptor_pthread_create /home/coder/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:243
    #1 0x7f11d6ae5758 in _PR_CreateThread /home/coder/OpenSrcCode/firefox/nsprpub/pr/src/pthreads/ptthread.c:457:14
    #2 0x7f11d6ae536a in PR_CreateThread /home/coder/OpenSrcCode/firefox/nsprpub/pr/src/pthreads/ptthread.c:548:12
    #3 0x7f11bc7d61a9 in nsThread::Init() /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:643:8
    #4 0x7f11bc7f589f in nsThreadManager::NewThread(unsigned int, unsigned int, nsIThread**) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThreadManager.cpp:260:17
    #5 0x7f11bc8654bd in NS_NewThread(nsIThread**, nsIRunnable*, unsigned int) /home/coder/OpenSrcCode/firefox/xpcom/glue/nsThreadUtils.cpp:83:5
    #6 0x7f11bd857b38 in NS_NewNamedThread<16> /home/coder/OpenSrcCode/firefox/objdir-ff-asan/dist/include/nsThreadUtils.h:82:17
    #7 0x7f11bd857b38 in (anonymous namespace)::ParentImpl::CreateBackgroundThread() /home/coder/OpenSrcCode/firefox/ipc/glue/BackgroundImpl.cpp:1142
    #8 0x7f11bd85e1c8 in CreateActorForSameProcess /home/coder/OpenSrcCode/firefox/ipc/glue/BackgroundImpl.cpp:1075:30
    #9 0x7f11bd85e1c8 in (anonymous namespace)::ChildImpl::OpenProtocolOnMainThread(nsIEventTarget*) /home/coder/OpenSrcCode/firefox/ipc/glue/BackgroundImpl.cpp:2020
    #10 0x7f11bd82bf6f in (anonymous namespace)::ChildImpl::GetOrCreateForCurrentThread(nsIIPCBackgroundChildCreateCallback*) /home/coder/OpenSrcCode/firefox/ipc/glue/BackgroundImpl.cpp:1643:9
    #11 0x7f11c2d3885d in ServiceWorkerManager /home/coder/OpenSrcCode/firefox/dom/workers/ServiceWorkerManager.cpp:235:3
    #12 0x7f11c2d3885d in mozilla::dom::workers::ServiceWorkerManager::GetInstance() /home/coder/OpenSrcCode/firefox/dom/workers/ServiceWorkerManager.cpp:1347
    #13 0x7f11bfa9d93c in nsDocument::SetScriptGlobalObject(nsIScriptGlobalObject*) /home/coder/OpenSrcCode/firefox/dom/base/nsDocument.cpp:4469:40
    #14 0x7f11c3ca5f3e in nsDocumentViewer::Close(nsISHEntry*) /home/coder/OpenSrcCode/firefox/layout/base/nsDocumentViewer.cpp:1509:7
    #15 0x7f11c4b96510 in nsDocShell::SetupNewViewer(nsIContentViewer*) /home/coder/OpenSrcCode/firefox/docshell/base/nsDocShell.cpp:9348:5
    #16 0x7f11c4b95159 in nsDocShell::Embed(nsIContentViewer*, char const*, nsISupports*) /home/coder/OpenSrcCode/firefox/docshell/base/nsDocShell.cpp:7231:17
    #17 0x7f11c4b2aca2 in nsDocShell::CreateContentViewer(nsACString_internal const&, nsIRequest*, nsIStreamListener**) /home/coder/OpenSrcCode/firefox/docshell/base/nsDocShell.cpp:9177:3
    #18 0x7f11c4b27e64 in nsDSURIContentListener::DoContent(nsACString_internal const&, bool, nsIRequest*, nsIStreamListener**, bool*) /home/coder/OpenSrcCode/firefox/docshell/base/nsDSURIContentListener.cpp:128:10
    #19 0x7f11bea8b5e4 in nsDocumentOpenInfo::TryContentListener(nsIURIContentListener*, nsIChannel*) /home/coder/OpenSrcCode/firefox/uriloader/base/nsURILoader.cpp:736:17
    #20 0x7f11bea87f8f in nsDocumentOpenInfo::DispatchContent(nsIRequest*, nsISupports*) /home/coder/OpenSrcCode/firefox/uriloader/base/nsURILoader.cpp:414:30
    #21 0x7f11bea86f9f in nsDocumentOpenInfo::OnStartRequest(nsIRequest*, nsISupports*) /home/coder/OpenSrcCode/firefox/uriloader/base/nsURILoader.cpp:277:8
    #22 0x7f11bc9720bb in nsBaseChannel::OnStartRequest(nsIRequest*, nsISupports*) /home/coder/OpenSrcCode/firefox/netwerk/base/nsBaseChannel.cpp:809:14
    #23 0x7f11bc9bc12e in nsInputStreamPump::OnStateStart() /home/coder/OpenSrcCode/firefox/netwerk/base/nsInputStreamPump.cpp:524:14
    #24 0x7f11bc9bb621 in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*) /home/coder/OpenSrcCode/firefox/netwerk/base/nsInputStreamPump.cpp:426:25
    #25 0x7f11bc796bcd in nsInputStreamReadyEvent::Run() /home/coder/OpenSrcCode/firefox/xpcom/io/nsStreamUtils.cpp:95:9
    #26 0x7f11bc7da84b in nsThread::ProcessNextEvent(bool, bool*) /home/coder/OpenSrcCode/firefox/xpcom/threads/nsThread.cpp:1213:7
    #27 0x7f11bc866888 in NS_ProcessNextEvent(nsIThread*, bool) /home/coder/OpenSrcCode/firefox/xpcom/glue/nsThreadUtils.cpp:361:10
    #28 0x7f11bd855651 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /home/coder/OpenSrcCode/firefox/ipc/glue/MessagePump.cpp:96:21
    #29 0x7f11bd748188 in RunInternal /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:232:3
    #30 0x7f11bd748188 in RunHandler /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:225
    #31 0x7f11bd748188 in MessageLoop::Run() /home/coder/OpenSrcCode/firefox/ipc/chromium/src/base/message_loop.cc:205
    #32 0x7f11c3381f4f in nsBaseAppShell::Run() /home/coder/OpenSrcCode/firefox/widget/nsBaseAppShell.cpp:156:3
    #33 0x7f11c55b1e81 in nsAppStartup::Run() /home/coder/OpenSrcCode/firefox/toolkit/components/startup/nsAppStartup.cpp:283:19
    #34 0x7f11c574397f in XREMain::XRE_mainRun() /home/coder/OpenSrcCode/firefox/toolkit/xre/nsAppRunner.cpp:4467:10
    #35 0x7f11c5744e2e in XREMain::XRE_main(int, char**, nsXREAppData const*) /home/coder/OpenSrcCode/firefox/toolkit/xre/nsAppRunner.cpp:4600:8
    #36 0x7f11c5745cfc in XRE_main /home/coder/OpenSrcCode/firefox/toolkit/xre/nsAppRunner.cpp:4691:16
    #37 0x4fc1e8 in do_main /home/coder/OpenSrcCode/firefox/browser/app/nsBrowserApp.cpp:328:10
    #38 0x4fc1e8 in main /home/coder/OpenSrcCode/firefox/browser/app/nsBrowserApp.cpp:461
    #39 0x7f11d5750ec4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4)

SUMMARY: AddressSanitizer: heap-use-after-free /home/coder/OpenSrcCode/firefox/ipc/glue/ProtocolUtils.cpp:61:43 in mozilla::ipc::IPCResult::Fail(mozilla::NotNull<mozilla::ipc::IProtocol*>, char const*, char const*)
Shadow bytes around the buggy address:
  0x0c22803618c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c22803618d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c22803618e0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c22803618f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2280361900: 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
=>0x0c2280361910: fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd
  0x0c2280361920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
  0x0c2280361930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2280361940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2280361950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2280361960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==5720==ABORTING

Comment 4

[Bevis Tseng[:bevis][:btseng](Exited)]

Update my finding so far:
1. The Use-After-Free was happened inside VersionChangeTransaction::RecvDeleteMe() right after PBackgroundIDBVersionChangeTransactionParent::Send__delete__(this) was done with negative result. [1]
2. No mater the result of Send__delete__(this) is OK or not, this actor will always be destroy inside this function. [2]
3. The *negative* IPCResult returned VersionChangeTransaction::RecvDeleteMe() always requires a vaild Actor pointer as input. [1]

So either this is design intention that caller has to figure out why can't we Send__delete__ message successfully in [3] when this symptom happened or this is a design issue in IPC that prevents us to return a negative IPCResult without providing a valid actor pointer as input.

Jan, do you have any suggestion on this?

Keep NI on me for further investigation.

I'll try to see if I can reproduce this locally to figure out why we can't see this __delete__ message correctly.

[1] http://searchfox.org/mozilla-central/rev/59bb309e38b10aba63dea8505fb800e99fe821d6/dom/indexedDB/ActorsParent.cpp#15931
[2] http://searchfox.org/mozilla-central/source/__GENERATED__/ipc/ipdl/PBackgroundIDBVersionChangeTransactionParent.cpp#107
[3] http://searchfox.org/mozilla-central/source/__GENERATED__/ipc/ipdl/PBackgroundIDBVersionChangeTransactionParent.cpp#103

Comment 5

[Bevis Tseng[:bevis][:btseng](Exited)]

Per offline discussion with :kanru, the UAF issue was caused by the design change in Bug 1314254.
In addition to what have been done in bug 1314254, we need to handle the case when the actor was destroyed before reporting an IPCResult::Fail() for Send__delete__().

CC :kanru for further comment on this bug.

Comment 6

[Andrew McCreight [:mccr8]]

[Tracking Requested - why for this release]:

Comment 7

[Marcia Knous [:marcia]]

Tracking 53+ for this sec crit bug.

Comment 8

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: Use actor's manager in case actor is already deleted.

I checked all Send__delete__(this) case and fixed them by using their manager actor.

I also checked other Send__delete__ usage and they don't uses the deleted actor afterwards. I'll audit other IPC_FAIL cases later.

Comment 9

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Comment on attachment 8813951 [details] [diff] [review]
Use actor's manager in case actor is already deleted.

Review of attachment 8813951 [details] [diff] [review]:
-----------------------------------------------------------------

I'm worried that we're going to introduce new cases like this without realizing it. What if we return early here if the Send call fails?
http://searchfox.org/mozilla-central/source/__GENERATED__/ipc/ipdl/PBackgroundFileHandleParent.cpp#82
Send will only fail if the connection has been closed. In that case we're going to tear down the actor very soon anyway.

Comment 10

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Bill McCloskey (:billm) from comment #9)
> Comment on attachment 8813951 [details] [diff] [review]
> Use actor's manager in case actor is already deleted.
> 
> Review of attachment 8813951 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I'm worried that we're going to introduce new cases like this without
> realizing it. What if we return early here if the Send call fails?
> http://searchfox.org/mozilla-central/source/__GENERATED__/ipc/ipdl/
> PBackgroundFileHandleParent.cpp#82
> Send will only fail if the connection has been closed. In that case we're
> going to tear down the actor very soon anyway.

Yeah, I think that deallocating in Send__delete__ is a footgun. I'm worried that early return might prevent some expected side effect in the actor destructor. But don't deallocate immediately if Send fails sounds like a saner choice.

Comment 11

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: Return early in Send__delete__ method when Send fails.

Tested with the repro script.

Comment 12

[Kan-Ru Chen [:kanru] (UTC+9)]

Attachment: Return early in Send__delete__ method when Send fails.

Update patch to include more context.

Comment 13

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Comment on attachment 8813969 [details] [diff] [review]
Return early in Send__delete__ method when Send fails.

Review of attachment 8813969 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not sure if the code for blocking __delete__ is correct. It seems like it was already broken before this patch. But we don't use that in the tree, so I think it's fine.

Thanks for the fix!

Comment 14

[Kan-Ru Chen [:kanru] (UTC+9)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a9856363aac76201b73e4885673889a4983e5f21

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/a9856363aac7

Comment 16

[Phil Ringnalda (:philor)]

Backed out in https://hg.mozilla.org/mozilla-central/rev/ee388da4aeb6 for a variety of Windows opt plugin test crashes, the most common (not obviously broken stack) flavors being https://treeherder.mozilla.org/logviewer.html#?job_id=39879993&repo=mozilla-inbound and https://treeherder.mozilla.org/logviewer.html#?job_id=39879956&repo=mozilla-inbound

Comment 17

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Phil Ringnalda (:philor) from comment #16)
> Backed out in https://hg.mozilla.org/mozilla-central/rev/ee388da4aeb6 for a
> variety of Windows opt plugin test crashes, the most common (not obviously
> broken stack) flavors being
> https://treeherder.mozilla.org/logviewer.html#?job_id=39879993&repo=mozilla-
> inbound and
> https://treeherder.mozilla.org/logviewer.html#?job_id=39879956&repo=mozilla-
> inbound

I'm still investigating this. I haven't been able to reproduce it locally. The two most common stacks looks totally unrelated stuff and unrelated to plugins. The crashing address is filled with 0xe5 so that indicates we have a UAF but I'm not sure how not freeing could lead to UAF.

Comment 18

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Kan-Ru Chen [:kanru] (UTC+8) from comment #17)
> (In reply to Phil Ringnalda (:philor) from comment #16)
> > Backed out in https://hg.mozilla.org/mozilla-central/rev/ee388da4aeb6 for a
> > variety of Windows opt plugin test crashes, the most common (not obviously
> > broken stack) flavors being
> > https://treeherder.mozilla.org/logviewer.html#?job_id=39879993&repo=mozilla-
> > inbound and
> > https://treeherder.mozilla.org/logviewer.html#?job_id=39879956&repo=mozilla-
> > inbound
> 
> I'm still investigating this. I haven't been able to reproduce it locally.
> The two most common stacks looks totally unrelated stuff and unrelated to
> plugins. The crashing address is filled with 0xe5 so that indicates we have
> a UAF but I'm not sure how not freeing could lead to UAF.

The problem is that when MessageChannel::Call timeout, we don't close the channel immediately. PluginModuleParent handles that by call CloseWithTimeout by itself but its subprotocol didn't do that.

In PluginInstanceParent::~PluginInstanceParent() we set |mNPP->pdata = nullptr| but mNPP might already been destroyed.

Full captured ASan report:

=================================================================
==10305==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000fd350 at pc 0x7f7f2dde2fba bp 0x7ffd172ac800 sp 0x7ffd172ac7f8
WRITE of size 8 at 0x60f0000fd350 thread T0 (Web Content)

###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

-----------------------------------------------------
Suppressions used:
  count      bytes template
    178       2294 libc.so
    252      15568 libfontconfig.so
      2       3032 libcairo.so
-----------------------------------------------------

    #0 0x7f7f2dde2fb9 in mozilla::plugins::PluginInstanceParent::~PluginInstanceParent() /home/kanru/mozilla/gecko/dom/plugins/ipc/PluginInstanceParent.cpp:150:21
    #1 0x7f7f2dde304d in mozilla::plugins::PluginInstanceParent::~PluginInstanceParent() /home/kanru/mozilla/gecko/dom/plugins/ipc/PluginInstanceParent.cpp:148:1
    #2 0x7f7f2ddfb02c in mozilla::plugins::PluginModuleParent::DeallocPPluginInstanceParent(mozilla::plugins::PPluginInstanceParent*) /home/kanru/mozilla/gecko/dom/plugins/ipc/PluginModuleParent.cpp:1675:5
    #3 0x7f7f28c749b6 in mozilla::plugins::PPluginModuleParent::DeallocSubtree() /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/ipc/ipdl/PPluginModuleParent.cpp:1667:13
    #4 0x7f7f28c740ca in mozilla::plugins::PPluginModuleParent::OnChannelClose() /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/ipc/ipdl/PPluginModuleParent.cpp:1598:5
    #5 0x7f7f2de0199e in mozilla::plugins::PluginModuleParent::DoShutdown(short*) /home/kanru/mozilla/gecko/dom/plugins/ipc/PluginModuleParent.cpp:2510:5
    #6 0x7f7f2de0199e in mozilla::plugins::PluginModuleParent::NP_Shutdown(short*) /home/kanru/mozilla/gecko/dom/plugins/ipc/PluginModuleParent.cpp:2486
    #7 0x7f7f2dd0881f in nsNPAPIPlugin::Shutdown() /home/kanru/mozilla/gecko/dom/plugins/base/nsNPAPIPlugin.cpp:357:3
    #8 0x7f7f2dd98b63 in nsPluginTag::TryUnloadPlugin(bool) /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginTags.cpp:727:5
    #9 0x7f7f2dd284c3 in nsPluginHost::OnPluginInstanceDestroyed(nsPluginTag*) /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginHost.cpp:749:7
    #10 0x7f7f2dd4297d in nsPluginHost::StopPluginInstance(nsNPAPIPluginInstance*) /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginHost.cpp:3463:7
    #11 0x7f7f2a9e74b8 in nsObjectLoadingContent::DoStopPlugin(nsPluginInstanceOwner*) /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:3085:5
    #12 0x7f7f2a9e7b08 in nsObjectLoadingContent::StopPluginInstance() /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:3135:3
    #13 0x7f7f2a9c95b0 in CheckPluginStopEvent::Run() /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:255:3
    #14 0x7f7f276189cb in nsThread::ProcessNextEvent(bool, bool*) /home/kanru/mozilla/gecko/xpcom/threads/nsThread.cpp:1213:7
    #15 0x7f7f276abf4c in NS_ProcessNextEvent(nsIThread*, bool) /home/kanru/mozilla/gecko/xpcom/glue/nsThreadUtils.cpp:361:54
    #16 0x7f7f2873b26f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /home/kanru/mozilla/gecko/ipc/glue/MessagePump.cpp:96:21
    #17 0x7f7f28616528 in MessageLoop::RunInternal() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:232:3
    #18 0x7f7f28616528 in MessageLoop::RunHandler() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:225
    #19 0x7f7f28616528 in MessageLoop::Run() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:205
    #20 0x7f7f2e7a2b0f in nsBaseAppShell::Run() /home/kanru/mozilla/gecko/widget/nsBaseAppShell.cpp:156:3
    #21 0x7f7f30d2e227 in XRE_RunAppShell /home/kanru/mozilla/gecko/toolkit/xre/nsEmbedFunctions.cpp:924:12
    #22 0x7f7f28616528 in MessageLoop::RunInternal() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:232:3
    #23 0x7f7f28616528 in MessageLoop::RunHandler() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:225
    #24 0x7f7f28616528 in MessageLoop::Run() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:205
    #25 0x7f7f30d2d65e in XRE_InitChildProcess /home/kanru/mozilla/gecko/toolkit/xre/nsEmbedFunctions.cpp:756:7
    #26 0x4fa243 in content_process_main(int, char**) /home/kanru/mozilla/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:115:19
    #27 0x4fa243 in main /home/kanru/mozilla/gecko/browser/app/nsBrowserApp.cpp:438
    #28 0x7f7f41d34b44 in __libc_start_main /build/glibc-daoqzt/glibc-2.19/csu/libc-start.c:287
    #29 0x41f885 in _start (/home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/bin/firefox+0x41f885)

0x60f0000fd350 is located 32 bytes inside of 176-byte region [0x60f0000fd330,0x60f0000fd3e0)
freed by thread T0 (Web Content) here:
    #0 0x4c52f0 in __interceptor_free (/home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/bin/firefox+0x4c52f0)
    #1 0x7f7f2dd6f01f in nsNPAPIPluginInstance::Release() /home/kanru/mozilla/gecko/dom/plugins/base/nsNPAPIPluginInstance.cpp:112:1049
    #2 0x7f7f2dd6f01f in mozilla::RefPtrTraits<nsNPAPIPluginInstance>::Release(nsNPAPIPluginInstance*) /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/RefPtr.h:40
    #3 0x7f7f2dd6f01f in RefPtr<nsNPAPIPluginInstance>::ConstRemovingRefPtrTraits<nsNPAPIPluginInstance>::Release(nsNPAPIPluginInstance*) /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/RefPtr.h:399
    #4 0x7f7f2dd6f01f in RefPtr<nsNPAPIPluginInstance>::~RefPtr() /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/RefPtr.h:78
    #5 0x7f7f2dd6f01f in nsPluginInstanceOwner::~nsPluginInstanceOwner() /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginInstanceOwner.cpp:397
    #6 0x7f7f2dd6f20d in nsPluginInstanceOwner::~nsPluginInstanceOwner() /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginInstanceOwner.cpp:374:1
    #7 0x7f7f2dd6f66b in nsPluginInstanceOwner::Release() /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginInstanceOwner.cpp:399:1049
    #8 0x7f7f2a9e7b10 in mozilla::RefPtrTraits<nsPluginInstanceOwner>::Release(nsPluginInstanceOwner*) /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/RefPtr.h:40:5
    #9 0x7f7f2a9e7b10 in RefPtr<nsPluginInstanceOwner>::ConstRemovingRefPtrTraits<nsPluginInstanceOwner>::Release(nsPluginInstanceOwner*) /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/RefPtr.h:399
    #10 0x7f7f2a9e7b10 in RefPtr<nsPluginInstanceOwner>::~RefPtr() /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/RefPtr.h:78
    #11 0x7f7f2a9e7b10 in nsObjectLoadingContent::StopPluginInstance() /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:3138
    #12 0x7f7f2a9c95b0 in CheckPluginStopEvent::Run() /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:255:3
    #13 0x7f7f276189cb in nsThread::ProcessNextEvent(bool, bool*) /home/kanru/mozilla/gecko/xpcom/threads/nsThread.cpp:1213:7
    #14 0x7f7f276abf4c in NS_ProcessNextEvent(nsIThread*, bool) /home/kanru/mozilla/gecko/xpcom/glue/nsThreadUtils.cpp:361:54
    #15 0x7f7f2873b26f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /home/kanru/mozilla/gecko/ipc/glue/MessagePump.cpp:96:21
    #16 0x7f7f28616528 in MessageLoop::RunInternal() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:232:3
    #17 0x7f7f28616528 in MessageLoop::RunHandler() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:225
    #18 0x7f7f28616528 in MessageLoop::Run() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:205
    #19 0x7f7f2e7a2b0f in nsBaseAppShell::Run() /home/kanru/mozilla/gecko/widget/nsBaseAppShell.cpp:156:3
    #20 0x7f7f30d2e227 in XRE_RunAppShell /home/kanru/mozilla/gecko/toolkit/xre/nsEmbedFunctions.cpp:924:12
    #21 0x7f7f28616528 in MessageLoop::RunInternal() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:232:3
    #22 0x7f7f28616528 in MessageLoop::RunHandler() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:225
    #23 0x7f7f28616528 in MessageLoop::Run() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:205
    #24 0x7f7f30d2d65e in XRE_InitChildProcess /home/kanru/mozilla/gecko/toolkit/xre/nsEmbedFunctions.cpp:756:7
    #25 0x4fa243 in content_process_main(int, char**) /home/kanru/mozilla/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:115:19
    #26 0x4fa243 in main /home/kanru/mozilla/gecko/browser/app/nsBrowserApp.cpp:438
    #27 0x7f7f41d34b44 in __libc_start_main /build/glibc-daoqzt/glibc-2.19/csu/libc-start.c:287

previously allocated by thread T0 (Web Content) here:
    #0 0x4c5608 in __interceptor_malloc (/home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/bin/firefox+0x4c5608)
    #1 0x4fb23d in moz_xmalloc /home/kanru/mozilla/gecko/memory/mozalloc/mozalloc.cpp:83:17
    #2 0x7f7f2dd2a134 in operator new(unsigned long) /home/kanru/mozilla/gecko/obj-x86_64-pc-linux-gnu/dist/include/mozilla/mozalloc.h:194:12
    #3 0x7f7f2dd2a134 in nsPluginHost::TrySetUpPluginInstance(nsACString_internal const&, nsIURI*, nsPluginInstanceOwner*) /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginHost.cpp:970
    #4 0x7f7f2dd2996a in nsPluginHost::SetUpPluginInstance(nsACString_internal const&, nsIURI*, nsPluginInstanceOwner*) /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginHost.cpp:903:17
    #5 0x7f7f2dd29164 in nsPluginHost::InstantiatePluginInstance(nsACString_internal const&, nsIURI*, nsObjectLoadingContent*, nsPluginInstanceOwner**) /home/kanru/mozilla/gecko/dom/plugins/base/nsPluginHost.cpp:835:8
    #6 0x7f7f2a9cdde3 in nsObjectLoadingContent::InstantiatePluginInstance(bool) /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:744:8
    #7 0x7f7f2a9e6bb1 in nsObjectLoadingContent::SyncStartPluginInstance() /home/kanru/mozilla/gecko/dom/base/nsObjectLoadingContent.cpp:2966:10
    #8 0x7f7f276189cb in nsThread::ProcessNextEvent(bool, bool*) /home/kanru/mozilla/gecko/xpcom/threads/nsThread.cpp:1213:7
    #9 0x7f7f276abf4c in NS_ProcessNextEvent(nsIThread*, bool) /home/kanru/mozilla/gecko/xpcom/glue/nsThreadUtils.cpp:361:54
    #10 0x7f7f2873b26f in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /home/kanru/mozilla/gecko/ipc/glue/MessagePump.cpp:96:21
    #11 0x7f7f28616528 in MessageLoop::RunInternal() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:232:3
    #12 0x7f7f28616528 in MessageLoop::RunHandler() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:225
    #13 0x7f7f28616528 in MessageLoop::Run() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:205
    #14 0x7f7f2e7a2b0f in nsBaseAppShell::Run() /home/kanru/mozilla/gecko/widget/nsBaseAppShell.cpp:156:3
    #15 0x7f7f30d2e227 in XRE_RunAppShell /home/kanru/mozilla/gecko/toolkit/xre/nsEmbedFunctions.cpp:924:12
    #16 0x7f7f28616528 in MessageLoop::RunInternal() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:232:3
    #17 0x7f7f28616528 in MessageLoop::RunHandler() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:225
    #18 0x7f7f28616528 in MessageLoop::Run() /home/kanru/mozilla/gecko/ipc/chromium/src/base/message_loop.cc:205
    #19 0x7f7f30d2d65e in XRE_InitChildProcess /home/kanru/mozilla/gecko/toolkit/xre/nsEmbedFunctions.cpp:756:7
    #20 0x4fa243 in content_process_main(int, char**) /home/kanru/mozilla/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:115:19
    #21 0x4fa243 in main /home/kanru/mozilla/gecko/browser/app/nsBrowserApp.cpp:438
    #22 0x7f7f41d34b44 in __libc_start_main /build/glibc-daoqzt/glibc-2.19/csu/libc-start.c:287

SUMMARY: AddressSanitizer: heap-use-after-free /home/kanru/mozilla/gecko/dom/plugins/ipc/PluginInstanceParent.cpp:150:21 in mozilla::plugins::PluginInstanceParent::~PluginInstanceParent()
Shadow bytes around the buggy address:
  0x0c1e80017a10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1e80017a20: fd fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
  0x0c1e80017a30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
  0x0c1e80017a40: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c1e80017a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
=>0x0c1e80017a60: fa fa fa fa fa fa fd fd fd fd[fd]fd fd fd fd fd
  0x0c1e80017a70: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c1e80017a80: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1e80017a90: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c1e80017aa0: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1e80017ab0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==10305==ABORTING

Comment 19

[Kan-Ru Chen [:kanru] (UTC+9)]

The dom/plugin code uses blocking __delete__ functions and depends on the correct deallocation order. The blocking version doesn't destroy the subtree right away when the blocking call times out. I'm not sure when will the async version destroy the subtree but if it also have some timing difference we might have a similar problem.

I think we can either land my first patch and treat the Send__delete__ same as free() function, checking them in static analysis later, or land my second patch but only for the async version.

WDYT?

Comment 20

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Let's land the second patch, but only for async __delete__.

Comment 21

[Kan-Ru Chen [:kanru] (UTC+9)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/28a55c1ec275c381b2dfd174c8847d8339d7f652

Comment 22

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/28a55c1ec275

Comment 23

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Comment on attachment 8813951 [details] [diff] [review]
Use actor's manager in case actor is already deleted.

After seeing bug 1316267, I decided that we should take this patch instead. Hope that's okay Kan-Ru.

Comment 24

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/398939c728aa3fe3ff785cfa57d4f8c0bb01cbf9
https://hg.mozilla.org/integration/mozilla-inbound/rev/80818c86802088550a7c0b66a4584ba1886c1a62

Comment 25

[Kan-Ru Chen [:kanru] (UTC+9)]

No problem, thanks for fixing this.

Comment 26

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/80818c868020

Comment 28

[Paul Silaghi, QA [:pauly]]

(In reply to Looben Yang from comment #0)
> Steps to reproduce: 
> 	1. Run server side script UAF_Fail_Repro.js in Node.js (node
> UAF_Fail_Repro.js).
> 	2. Enter http://localhost:12345 in Firefox browser.
> 
> 
> Firefox version: 53.0a1 (2016-11-21) (32-bit)
> OS: Windows 10
> 
> 
> Stack trace:
> 
> 	(5bac.4fb4): Access violation - code c0000005 (!!! second chance !!!)
> 	eax=e5e5e5e5 ebx=23239dc0 ecx=0e1c2290 edx=1163f41c esi=14a3f82f
> edi=23239dc0
How exactly did you get this stack trace?
I'm trying to reproduce the initial problem, but I don't see anything like this.
I got a stack trace with WinDbg and also tried on a debug build 53.0a1 (2016-11-21), but without any luck.

Comment 29

[Looben Yang]

(In reply to Paul Silaghi, QA [:pauly] from comment #28)
> (In reply to Looben Yang from comment #0)
> > Steps to reproduce: 
> > 	1. Run server side script UAF_Fail_Repro.js in Node.js (node
> > UAF_Fail_Repro.js).
> > 	2. Enter http://localhost:12345 in Firefox browser.
> > 
> > 
> > Firefox version: 53.0a1 (2016-11-21) (32-bit)
> > OS: Windows 10
> > 
> > 
> > Stack trace:
> > 
> > 	(5bac.4fb4): Access violation - code c0000005 (!!! second chance !!!)
> > 	eax=e5e5e5e5 ebx=23239dc0 ecx=0e1c2290 edx=1163f41c esi=14a3f82f
> > edi=23239dc0
> How exactly did you get this stack trace?
> I'm trying to reproduce the initial problem, but I don't see anything like
> this.
> I got a stack trace with WinDbg and also tried on a debug build 53.0a1
> (2016-11-21), but without any luck.

I used the instructions in https://bugzilla.mozilla.org/show_bug.cgi?id=1319271#c0 to trigger the bug.
Probably you just need to let the test case running for a while. I remember it's not triggered instantly with this test case.