1226477 - Intermittent e10s PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::ipc::BackgroundChildImpl::ProcessingError(mozilla::ipc::HasResultCodes::Result, char const*)]

Status: RESOLVED WORKSFORME

(Core :: IPC, defect, P2)

Description

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/logviewer.html#?job_id=17565500&repo=mozilla-inbound

 14:33:02     INFO -  1361 INFO TEST-OK | dom/base/test/test_xhr_withCredentials.html | took 273ms
 14:33:02     INFO -  ++DOMWINDOW == 17 (0x7fd1a3b41c00) [pid = 4477] [serial = 2299] [outer = 0x7fd1a463a000]
 14:33:02     INFO -  ++DOMWINDOW == 18 (0x7fd1a3b42c00) [pid = 4477] [serial = 2300] [outer = 0x7fd1a463a000]
 14:33:02     INFO -  [Parent 4423] WARNING: '!aObserver', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/xpcom/ds/nsObserverService.cpp, line 261
 14:33:02     INFO -  1447972382333	addons.xpi	WARN	Exception running bootstrap method shutdown on [e-mail]: [Exception... "Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIObserverService.removeObserver]"  nsresult: "0x80070057 (NS_ERROR_ILLEGAL_VALUE)"  location: "JS frame :: resource://gre/modules/RemoteAddonsParent.jsm :: ObserverInterposition.methods.removeObserver :: line 388"  data: no] Stack trace: ObserverInterposition.methods.removeObserver()@resource://gre/modules/RemoteAddonsParent.jsm:388 < AddonInterpositionService.prototype.interposeProperty/desc.value()@resource://gre/components/multiprocessShims.js:160 < SpecialPowersObserver.prototype.uninit/<()@SpecialPowersObserver.jsm:149 < SpecialPowersObserver.prototype.uninit()@SpecialPowersObserver.jsm:148 < shutdown()@resource://gre/modules/addons/XPIProvider.jsm -> file:///tmp/tmpHX0Cmm.mozrunner/extensions/[e-mail]/bootstrap.js:35 < XPI_callBootstrapMethod()@resource://gre/modules/addons/XPIProvider.jsm:4475 < shutdownObserver()@resource://gre/modules/addons/XPIProvider.jsm:2716 < SpecialPowersObserver.prototype.receiveMessage()@SpecialPowersObserver.jsm:260 < permitUnload()@remote-browser.xml:322 < CanCloseWindow()@browser.js:17083 < canClose()@browser.js:15805 < SpecialPowersObserver.prototype.receiveMessage()@SpecialPowersObserver.jsm:260
 14:33:02     INFO -  --DOCSHELL 0x7fa7614dc800 == 6 [pid = 4423] [id = 6]
 14:33:02     INFO -  JavaScript error: resource://gre/modules/PerformanceStats.jsm, line 208: NS_ERROR_NOT_AVAILABLE: Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsIPerformanceStatsService.isMonitoringJank]
 14:33:02     INFO -  ###!!! [Parent][OnMaybeDequeueOne] Error: Channel closing: too late to send/recv, messages will be lost
 14:33:03     INFO -  --DOCSHELL 0x7fa77a423800 == 5 [pid = 4423] [id = 1]
 14:33:04     INFO -  --DOCSHELL 0x7fa76324a800 == 4 [pid = 4423] [id = 7]
 14:33:04     INFO -  --DOCSHELL 0x7fa76af82800 == 3 [pid = 4423] [id = 3]
 14:33:04     INFO -  --DOCSHELL 0x7fa7685d7000 == 2 [pid = 4423] [id = 12]
 14:33:04     INFO -  --DOCSHELL 0x7fa76af83000 == 1 [pid = 4423] [id = 4]
 14:33:05     INFO -  ]: --DOCSHELL 0x7fd1980bd000 == 4 [pid = 4477] [id = 95]
 14:33:05     INFO -  ###!!! [Child][MessageChannel] Error: (msgtype=0x9E0027,name=PNecko::Msg_RemoveSchedulingContext) Closed channel: cannot send/recv
 14:33:05     INFO -  [Child 4477] WARNING: MsgDropped in ContentChild: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/dom/ipc/ContentChild.cpp, line 2213
 14:33:05     INFO -  --DOCSHELL 0x7fd195a1d800 == 3 [pid = 4477] [id = 291]
 14:33:05     INFO -  ###!!! [Child][MessageChannel] Error: (msgtype=0x9E0027,name=PNecko::Msg_RemoveSchedulingContext) Closed channel: cannot send/recv
 14:33:05     INFO -  [Child 4477] WARNING: MsgDropped in ContentChild: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/dom/ipc/ContentChild.cpp, line 2213
 14:33:05     INFO -  --DOCSHELL 0x7fd1980ba800 == 2 [pid = 4477] [id = 343]
 14:33:05     INFO -  ###!!! [Child][MessageChannel] Error: (msgtype=0x9E0027,name=PNecko::Msg_RemoveSchedulingContext) Closed channel: cannot send/recv
 14:33:05     INFO -  [Child 4477] WARNING: MsgDropped in ContentChild: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/dom/ipc/ContentChild.cpp, line 2213
 14:33:05     INFO -  JavaScript error: resource://gre/modules/PerformanceStats.jsm, line 492: Error: forget() called twice
 14:33:05     INFO -  ###!!! [Parent][OnMaybeDequeueOne] Error: Channel closing: too late to send/recv, messages will be lost
 14:33:05     INFO -  [Child 4477] WARNING: NS_ENSURE_TRUE(context) failed: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/xpcom/threads/nsThread.cpp, line 769
 14:33:05     INFO -  ###!!! [Child][MessageChannel] Error: (msgtype=0x1E0001,name=PBlob::Msg___delete__) Closed channel: cannot send/recv
 14:33:05     INFO -  Hit MOZ_CRASH(MsgDropped) at /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/BackgroundChildImpl.cpp:134
 14:33:05     INFO -  #01: mozilla::ipc::MessageChannel::ReportConnectionError(char const*, IPC::Message*) const [xpcom/glue/Monitor.h:121]
 14:33:05     INFO -  #02: mozilla::ipc::MessageChannel::Send(IPC::Message*) [ipc/glue/MessageChannel.cpp:558]
 14:33:05     INFO -  #03: mozilla::dom::PBlobChild::Send__delete__(mozilla::dom::PBlobChild*) [obj-firefox/ipc/ipdl/PBlobChild.cpp:85]
 14:33:05     INFO -  #04: mozilla::dom::BlobChild::NoteDyingRemoteBlobImpl() [dom/ipc/Blob.cpp:3547]
 14:33:05     INFO -  #05: mozilla::dom::BlobChild::RemoteBlobImpl::Destroy() [dom/ipc/Blob.cpp:2214]
 14:33:05     INFO -  #06: mozilla::dom::BlobChild::RemoteBlobImpl::Release() [dom/ipc/Blob.cpp:2233]
 14:33:05     INFO -  #07: mozilla::dom::BlobChild::RemoteBlobSliceImpl::~RemoteBlobSliceImpl() [dom/ipc/Blob.cpp:1952]
 14:33:05     INFO -  #08: mozilla::dom::BlobChild::RemoteBlobSliceImpl::~RemoteBlobSliceImpl() [dom/ipc/Blob.cpp:1952]
 14:33:05     INFO -  #09: mozilla::dom::BlobChild::RemoteBlobImpl::Destroy() [dom/ipc/Blob.cpp:2214]
 14:33:05     INFO -  #10: mozilla::dom::BlobChild::RemoteBlobImpl::Release() [dom/ipc/Blob.cpp:2233]
 14:33:05     INFO -  #11: mozilla::dom::BlobChild::RemoteBlobSliceImpl::Release() [dom/ipc/Blob.cpp:2591]
 14:33:05     INFO -  #12: mozilla::dom::BlobChild::ActorDestroy(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) [dom/ipc/Blob.cpp:3572]
 14:33:05     INFO -  #13: mozilla::dom::PBlobChild::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) [obj-firefox/ipc/ipdl/PBlobChild.cpp:612]
 14:33:05     INFO -  #14: mozilla::ipc::PBackgroundChild::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) [obj-firefox/ipc/ipdl/PBackgroundChild.cpp:2000]
 14:33:05     INFO -  #15: mozilla::ipc::PBackgroundChild::OnChannelClose() [obj-firefox/ipc/ipdl/PBackgroundChild.cpp:1882]
 14:33:05     INFO -  #16: mozilla::ipc::MessageChannel::NotifyChannelClosed() [ipc/glue/MessageChannel.cpp:1953]
 14:33:05     INFO -  #17: mozilla::ipc::MessageChannel::Close() [ipc/glue/MessageChannel.cpp:1937]
 14:33:05     INFO -  #18: ChildImpl::ThreadLocalDestructor [ipc/glue/BackgroundImpl.cpp:420]
 14:33:05     INFO -  #19: PR_SetThreadPrivate [nsprpub/pr/src/threads/prtpd.c:184]
 14:33:05     INFO -  #20: ChildImpl::Shutdown [ipc/glue/BackgroundImpl.cpp:1626]
 14:33:05     INFO -  #21: ChildImpl::ShutdownObserver::Observe [ipc/glue/BackgroundImpl.cpp:2114]
 14:33:05     INFO -  #22: nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) [xpcom/glue/nsTArray.h:358]
 14:33:05     INFO -  #23: nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) [xpcom/glue/nsTHashtable.h:134]
 14:33:05     INFO -  #24: mozilla::ShutdownXPCOM(nsIServiceManager*) [xpcom/build/XPCOMInit.cpp:870]
 14:33:05     INFO -  #25: XRE_TermEmbedding [toolkit/xre/nsEmbedFunctions.cpp:210]
 14:33:05     INFO -  #26: mozilla::ipc::ScopedXREEmbed::Stop() [ipc/glue/ScopedXREEmbed.cpp:116]
 14:33:05     INFO -  #27: XRE_InitChildProcess [toolkit/xre/nsEmbedFunctions.cpp:628]
 14:33:05     INFO -  #28: content_process_main(int, char**) [ipc/contentproc/plugin-container.cpp:240]
 14:33:05     INFO -  #29: libc.so.6 + 0x2176d
 14:33:05     INFO -  #30: _start
 14:33:06     INFO -  ###!!! [Parent][OnMaybeDequeueOne] Error: Channel closing: too late to send/recv, messages will be lost
 14:33:06     INFO -  ###!!! [Parent][OnMaybeDequeueOne] Error: Channel closing: too late to send/recv, messages will be lost
 14:33:06     INFO -  ###!!! [Parent][OnMaybeDequeueOne] Error: Channel closing: too late to send/recv, messages will be lost
 14:33:06     INFO -  [Parent 4423] WARNING: NS_ENSURE_TRUE(context) failed: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/xpcom/threads/nsThread.cpp, line 769
 14:33:07     INFO -  --DOCSHELL 0x7fa76e5a0800 == 0 [pid = 4423] [id = 2]
 14:33:08     INFO -  --DOMWINDOW == 13 (0x7fa76878e800) [pid = 4423] [serial = 10] [outer = 0x7fa76ac53400] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 12 (0x7fa76878f000) [pid = 4423] [serial = 11] [outer = 0x7fa76ac53c00] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 11 (0x7fa76ac53400) [pid = 4423] [serial = 6] [outer = (nil)] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 10 (0x7fa76ac53c00) [pid = 4423] [serial = 7] [outer = (nil)] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 9 (0x7fa76e5d8800) [pid = 4423] [serial = 4] [outer = (nil)] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 8 (0x7fa774da0c00) [pid = 4423] [serial = 28] [outer = (nil)] [url = about:newtab]
 14:33:08     INFO -  --DOMWINDOW == 7 (0x7fa77a404c00) [pid = 4423] [serial = 30] [outer = (nil)] [url = about:newtab]
 14:33:08     INFO -  --DOMWINDOW == 6 (0x7fa761509400) [pid = 4423] [serial = 15] [outer = (nil)] [url = data:application/vnd.mozilla.xul+xml;charset=utf-8,<window%20id='win'/>]
 14:33:08     INFO -  --DOMWINDOW == 5 (0x7fa76e5d7c00) [pid = 4423] [serial = 3] [outer = (nil)] [url = chrome://browser/content/browser.xul]
 14:33:08     INFO -  --DOMWINDOW == 4 (0x7fa761351800) [pid = 4423] [serial = 16] [outer = (nil)] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 3 (0x7fa76459e000) [pid = 4423] [serial = 17] [outer = (nil)] [url = about:blank]
 14:33:08     INFO -  --DOMWINDOW == 2 (0x7fa761359800) [pid = 4423] [serial = 13] [outer = (nil)] [url = data:application/vnd.mozilla.xul+xml;charset=utf-8,<window%20id='win'/>]
 14:33:08     INFO -  --DOMWINDOW == 1 (0x7fa77a493400) [pid = 4423] [serial = 1] [outer = (nil)] [url = resource://gre-resources/hiddenWindow.html]
 14:33:08     INFO -  --DOMWINDOW == 0 (0x7fa76dd0ec00) [pid = 4423] [serial = 5] [outer = (nil)] [url = resource://gre-resources/hiddenWindow.html]
 14:33:09     INFO -  nsStringStats
 14:33:09     INFO -   => mAllocCount:        1140985
 14:33:09     INFO -   => mReallocCount:       133995
 14:33:09     INFO -   => mFreeCount:         1140985
 14:33:09     INFO -   => mShareCount:         822020
 14:33:09     INFO -   => mAdoptCount:          31414
 14:33:09     INFO -   => mAdoptFreeCount:      31414
 14:33:09     INFO -   => Process ID: 4423, Thread ID: 140357823522624
 14:33:09     INFO -  TEST-INFO | Main app process: exit 0
 14:33:09     INFO -  runtests.py | Application ran for: 0:16:28.668021
 14:33:09     INFO -  zombiecheck | Reading PID log: /tmp/tmp8GsVg8pidlog
 14:33:09     INFO -  ==> process 4423 launched child process 4477
 14:33:09     INFO -  ==> process 4423 launched child process 4799
 14:33:09     INFO -  zombiecheck | Checking for orphan process with PID: 4477
 14:33:09     INFO -  zombiecheck | Checking for orphan process with PID: 4799
 14:33:23     INFO -  mozcrash Saved minidump as /builds/slave/test/build/blobber_upload_dir/3575ca42-d0ae-ba76-5ba336f3-2150ff4b.dmp
 14:33:23     INFO -  mozcrash Saved app info as /builds/slave/test/build/blobber_upload_dir/3575ca42-d0ae-ba76-5ba336f3-2150ff4b.extra
 14:33:23  WARNING -  PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::ipc::BackgroundChildImpl::ProcessingError(mozilla::ipc::HasResultCodes::Result, char const*)]
 14:33:23     INFO -  Crash dump filename: /tmp/tmpHX0Cmm.mozrunner/minidumps/3575ca42-d0ae-ba76-5ba336f3-2150ff4b.dmp
 14:33:23     INFO -  Operating system: Linux
 14:33:23     INFO -                    0.0.0 Linux 3.2.0-76-generic #111-Ubuntu SMP Tue Jan 13 22:16:09 UTC 2015 x86_64
 14:33:23     INFO -  CPU: amd64
 14:33:23     INFO -       family 6 model 62 stepping 4
 14:33:23     INFO -       1 CPU
 14:33:23     INFO -  Crash reason:  SIGSEGV
 14:33:23     INFO -  Crash address: 0x0
 14:33:23     INFO -  Process uptime: not available
 14:33:23     INFO -  Thread 0 (crashed)
 14:33:23     INFO -   0  libxul.so!mozilla::ipc::BackgroundChildImpl::ProcessingError(mozilla::ipc::HasResultCodes::Result, char const*) [BackgroundChildImpl.cpp:938041e183e5 : 134 + 0x15]
 14:33:23     INFO -      rax = 0x0000000000000000   rdx = 0x0000000000000000
 14:33:23     INFO -      rcx = 0x00007fd1b9d8af4d   rbx = 0x00007fff309d8860
 14:33:23     INFO -      rsi = 0x00007fd1ba05ea80   rdi = 0x00007fd1ba05d180
 14:33:23     INFO -      rbp = 0x00007fff309d88e0   rsp = 0x00007fff309d8860
 14:33:23     INFO -       r8 = 0x00007fd1c17dd9c0    r9 = 0x6c702f786f666572
 14:33:23     INFO -      r10 = 0x6e6f632d6e696775   r11 = 0x0000000000000000
 14:33:23     INFO -      r12 = 0x00007fff309d8880   r13 = 0x0000000000000001
 14:33:23     INFO -      r14 = 0x00007fd1a5a47878   r15 = 0x00007fd1be647756
 14:33:23     INFO -      rip = 0x00007fd1bb9d161e
 14:33:23     INFO -      Found by: given as instruction pointer in context
 14:33:23     INFO -   1  libxul.so!mozilla::ipc::MessageChannel::ReportConnectionError(char const*, IPC::Message*) const [MessageChannel.cpp:938041e183e5 : 1704 + 0x19]
 14:33:23     INFO -      rbx = 0x00007fd1be644dee   rbp = 0x00007fff309d8b30
 14:33:23     INFO -      rsp = 0x00007fff309d88f0   r12 = 0x00007fd1a5a47868
 14:33:23     INFO -      r13 = 0x00007fd1a5a6e6a0   r14 = 0x00007fd1a5a47878
 14:33:23     INFO -      r15 = 0x00007fd1be647756   rip = 0x00007fd1bb9dbad2
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   2  libxul.so!mozilla::ipc::MessageChannel::Send(IPC::Message*) [MessageChannel.cpp:938041e183e5 : 558 + 0x5]
 14:33:23     INFO -      rbx = 0x0000000000000000   rbp = 0x00007fff309d8b70
 14:33:23     INFO -      rsp = 0x00007fff309d8b40   r12 = 0x00007fd1a5a47878
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bb9e8c71
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   3  libxul.so!mozilla::dom::PBlobChild::Send__delete__(mozilla::dom::PBlobChild*) [PBlobChild.cpp:938041e183e5 : 83 + 0xc]
 14:33:23     INFO -      rbx = 0x00007fd1944f7e80   rbp = 0x00007fff309d8bc0
 14:33:23     INFO -      rsp = 0x00007fff309d8b80   r12 = 0x00007fd1943bd070
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bbb872fe
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   4  libxul.so!mozilla::dom::BlobChild::NoteDyingRemoteBlobImpl() [Blob.cpp:938041e183e5 : 3546 + 0x8]
 14:33:23     INFO -      rbx = 0x00007fd1944f7e80   rbp = 0x00007fff309d8c00
 14:33:23     INFO -      rsp = 0x00007fff309d8bd0   r12 = 0x00007fd1944f7f28
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddd2df
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   5  libxul.so!mozilla::dom::BlobChild::RemoteBlobImpl::Destroy() [Blob.cpp:938041e183e5 : 2211 + 0x9]
 14:33:23     INFO -      rbx = 0x00007fd1944f7f20   rbp = 0x00007fff309d8c40
 14:33:23     INFO -      rsp = 0x00007fff309d8c10   r12 = 0x00007fd1944f7f28
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddd527
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   6  libxul.so!mozilla::dom::BlobChild::RemoteBlobImpl::Release() [Blob.cpp:938041e183e5 : 2233 + 0x3a]
 14:33:23     INFO -      rbx = 0x00007fd1944f7f20   rbp = 0x00007fff309d8c70
 14:33:23     INFO -      rsp = 0x00007fff309d8c50   r12 = 0x00007fd1944f7f28
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddd6ec
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   7  libxul.so!mozilla::dom::BlobChild::RemoteBlobSliceImpl::~RemoteBlobSliceImpl() [Blob.cpp:938041e183e5 : 1952 + 0x10]
 14:33:23     INFO -      rbx = 0x00007fd1980833a0   rbp = 0x00007fff309d8c90
 14:33:23     INFO -      rsp = 0x00007fff309d8c80   r12 = 0x00007fd1980833a8
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddc1b4
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   8  libxul.so!mozilla::dom::BlobChild::RemoteBlobSliceImpl::~RemoteBlobSliceImpl() [Blob.cpp:938041e183e5 : 1952 + 0x5]
 14:33:23     INFO -      rbx = 0x00007fd1980833a0   rbp = 0x00007fff309d8cb0
 14:33:23     INFO -      rsp = 0x00007fff309d8ca0   r12 = 0x00007fd1980833a8
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddc1ce
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -   9  libxul.so!mozilla::dom::BlobChild::RemoteBlobImpl::Destroy() [Blob.cpp:938041e183e5 : 2214 + 0xc]
 14:33:23     INFO -      rbx = 0x00007fd1980833a0   rbp = 0x00007fff309d8cf0
 14:33:23     INFO -      rsp = 0x00007fff309d8cc0   r12 = 0x00007fd1980833a8
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddd533
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  10  libxul.so!mozilla::dom::BlobChild::RemoteBlobImpl::Release() [Blob.cpp:938041e183e5 : 2233 + 0x3a]
 14:33:23     INFO -      rbx = 0x00007fd1980833a0   rbp = 0x00007fff309d8d20
 14:33:23     INFO -      rsp = 0x00007fff309d8d00   r12 = 0x00007fd1980833a8
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddd6ec
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  11  libxul.so!mozilla::dom::BlobChild::RemoteBlobSliceImpl::Release() [Blob.cpp:938041e183e5 : 2591 + 0x5]
 14:33:23     INFO -      rbx = 0x00007fd1980833a0   rbp = 0x00007fff309d8d50
 14:33:23     INFO -      rsp = 0x00007fff309d8d30   r12 = 0x0000000000000000
 14:33:23     INFO -      r13 = 0x0000000000000003   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddd711
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  12  libxul.so!mozilla::dom::BlobChild::ActorDestroy(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) [Blob.cpp:938041e183e5 : 3565 + 0x6]
 14:33:23     INFO -      rbx = 0x00007fd197c8f420   rbp = 0x00007fff309d8d70
 14:33:23     INFO -      rsp = 0x00007fff309d8d60   r12 = 0x0000000000000000
 14:33:23     INFO -      r13 = 0x0000000000000003   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bcddfff9
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  13  libxul.so!mozilla::dom::PBlobChild::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) [PBlobChild.cpp:938041e183e5 : 611 + 0xf]
 14:33:23     INFO -      rbx = 0x00007fd197c8f420   rbp = 0x00007fff309d8dc0
 14:33:23     INFO -      rsp = 0x00007fff309d8d80   r12 = 0x0000000000000000
 14:33:23     INFO -      r13 = 0x0000000000000003   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x00007fff309d8d88   rip = 0x00007fd1bbb8706b
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  14  libxul.so!mozilla::ipc::PBackgroundChild::DestroySubtree(mozilla::ipc::IProtocolManager<mozilla::ipc::IProtocol>::ActorDestroyReason) [PBackgroundChild.cpp:938041e183e5 : 2001 + 0x5]
 14:33:23     INFO -      rbx = 0x00007fd1a5a47800   rbp = 0x00007fff309d8e10
 14:33:23     INFO -      rsp = 0x00007fff309d8dd0   r12 = 0x0000000000000003
 14:33:23     INFO -      r13 = 0x0000000000000016   r14 = 0x0000000000000003
 14:33:23     INFO -      r15 = 0x0000000000000000   rip = 0x00007fd1bba14a84
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  15  libxul.so!mozilla::ipc::PBackgroundChild::OnChannelClose() [PBackgroundChild.cpp:938041e183e5 : 1881 + 0x5]
 14:33:23     INFO -      rbx = 0x00007fd1a5a47800   rbp = 0x00007fff309d8e30
 14:33:23     INFO -      rsp = 0x00007fff309d8e20   r12 = 0x00007fd1943bd070
 14:33:23     INFO -      r13 = 0x00007fd1bb9da77e   r14 = 0x0000000000000020
 14:33:23     INFO -      r15 = 0x0000000000000000   rip = 0x00007fd1bba14e75
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  16  libxul.so!mozilla::ipc::MessageChannel::NotifyChannelClosed() [MessageChannel.cpp:938041e183e5 : 1950 + 0x11]
 14:33:23     INFO -      rbx = 0x00007fd1a5a47868   rbp = 0x00007fff309d8e50
 14:33:23     INFO -      rsp = 0x00007fff309d8e40   r12 = 0x00007fd1943bd070
 14:33:23     INFO -      r13 = 0x00007fd1bb9da77e   r14 = 0x0000000000000020
 14:33:23     INFO -      r15 = 0x0000000000000000   rip = 0x00007fd1bb9e91e6
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  17  libxul.so!mozilla::ipc::MessageChannel::Close() [MessageChannel.cpp:938041e183e5 : 1937 + 0x8]
 14:33:23     INFO -      rbx = 0x00007fd1a5a47868   rbp = 0x00007fff309d8e90
 14:33:23     INFO -      rsp = 0x00007fff309d8e60   r12 = 0x00007fd1943bd070
 14:33:23     INFO -      r13 = 0x00007fd1bb9da77e   r14 = 0x0000000000000020
 14:33:23     INFO -      r15 = 0x0000000000000000   rip = 0x00007fd1bb9e9363
 14:33:23     INFO -      Found by: call frame info
 14:33:23     INFO -  18  libxul.so!ChildImpl::ThreadLocalDestructor [BackgroundImpl.cpp:938041e183e5 : 419 + 0xd]
 14:33:23     INFO -      rbx = 0x00007fd1a9046260   rbp = 0x00007fff309d8ed0
 14:33:23     INFO -      rsp = 0x00007fff309d8ea0   r12 = 0x0000000000000004
 14:33:23     INFO -      r13 = 0x0000000000000000   r14 = 0x0000000000000020
 14:33:23     INFO -      r15 = 0x0000000000000000   rip = 0x00007fd1bb9e243f
 14:33:23     INFO -      Found by: call frame info
...

Comment 1

[Intermittent Failures Robot]

7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* fx-team: 1
* b2g-inbound: 1

Platform breakdown:
* linux64: 4
* linux32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-11-16&endday=2015-11-22&tree=all

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Kyle, just looking at the stack, this looks like something worth fixing. Do you know who works on blob stuff nowadays?

Comment 3

[Andrew McCreight [:mccr8]]

I think baku works on blob stuff the most.

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Andrea, or me on sufficiently long time scales.

Comment 5

[Andrea Marchesini [:baku]]

This error has been introduced by bug 1211266.

Comment 6

[Phil Ringnalda (:philor)]

This crash, which I filed on November 19th, was caused by a patch which you pushed on November 24th? That strikes me as unlikely.

Comment 7

[Intermittent Failures Robot]

19 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 9
* mozilla-inbound: 6
* mozilla-central: 2
* try: 1
* b2g-inbound: 1

Platform breakdown:
* linux64: 15
* linux32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-11-23&endday=2015-11-29&tree=all

Comment 8

[Intermittent Failures Robot]

29 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 16
* try: 4
* mozilla-central: 3
* fx-team: 3
* b2g-inbound: 3

Platform breakdown:
* linux64: 21
* linux32: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-11-30&endday=2015-12-06&tree=all

Comment 9

[Intermittent Failures Robot]

11 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* fx-team: 3
* b2g-inbound: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 10
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-12-07&endday=2015-12-13&tree=all

Comment 10

[Intermittent Failures Robot]

17 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* try: 2
* fx-team: 2
* b2g-inbound: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 11
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-12-14&endday=2015-12-20&tree=all

Comment 11

[Intermittent Failures Robot]

19 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 8
* mozilla-inbound: 7
* mozilla-aurora: 2
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* linux64: 12
* linux32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-12-21&endday=2015-12-27&tree=all

Comment 12

[Intermittent Failures Robot]

16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* mozilla-aurora: 4
* b2g-inbound: 4
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* linux32: 9
* linux64: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2015-12-28&endday=2016-01-03&tree=all

Comment 13

[Intermittent Failures Robot]

21 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 6
* mozilla-inbound: 6
* fx-team: 4
* mozilla-central: 3
* mozilla-aurora: 1
* b2g-inbound: 1

Platform breakdown:
* linux32: 11
* linux64: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-01-04&endday=2016-01-10&tree=all

Comment 14

[Intermittent Failures Robot]

28 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 11
* mozilla-central: 5
* fx-team: 4
* try: 3
* b2g-inbound: 3
* mozilla-aurora: 2

Platform breakdown:
* linux64: 19
* linux32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-01-11&endday=2016-01-17&tree=all

Comment 15

[Intermittent Failures Robot]

34 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 17
* try: 8
* mozilla-central: 3
* fx-team: 3
* b2g-inbound: 2
* mozilla-aurora: 1

Platform breakdown:
* linux64: 24
* linux32: 9
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-01-18&endday=2016-01-24&tree=all

Comment 16

[Intermittent Failures Robot]

23 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* fx-team: 7
* mozilla-central: 3
* try: 2
* mozilla-aurora: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 14
* linux32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-01-25&endday=2016-01-31&tree=all

Comment 17

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Do you actually understand the problem here?  It looks like the link between the two processes has been shut down, and ContentChild just drops all messages while BackgroundChild asserts and dies.  Both protocols are getting the same errors (MsgDropped).

Comment 18

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Yeah, that's pretty much it. We're not supposed to send any messages after ActorDestroy is called, so the Send__delete__ call here is wrong.

We are pretty lax about this, which is why ContentChild does what it does. But I think bad stuff can still happen in some cases. If you do the Send operation on a non-toplevel actor after the connection is closed, then I don't think there's any guarantee that the MessageChannel will still be alive, so it could be a UAF.

In this case we may be safe, assuming the Send__delete__ operation always happens synchronously with ActorDestroy. But I'd rather fix this case and hold the line at PBackground rather than loosen our assertions there as well.

Comment 19

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Ok ... I'm fine with holding the line.  I don't really understand what went wrong to get us in this situation though :(

Comment 20

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I think it went something like this:
- We used to crash in ContentChild as well.
- As a consequence, the content process always crashed on shutdown. Nobody paid much attention because, at this point, neither b2g nor e10s had very good test coverage.
- We wanted to turn on leak checking for content processes, so we either had to fix all the broken message sends or turn the crash into a warning. At this point there were too many to fix.

Comment 21

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Oh, I meant this specific bug :P  I can totally envision how we got here organizationally.

Comment 22

[Intermittent Failures Robot]

37 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 17
* fx-team: 12
* try: 4
* mozilla-aurora: 3
* mozilla-central: 1

Platform breakdown:
* linux64: 22
* linux32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-02-01&endday=2016-02-07&tree=all

Comment 23

[Intermittent Failures Robot]

21 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* mozilla-central: 4
* fx-team: 4
* try: 3
* mozilla-beta: 3
* mozilla-aurora: 2

Platform breakdown:
* linux64: 13
* linux32: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-02-08&endday=2016-02-14&tree=all

Comment 24

[Intermittent Failures Robot]

41 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 19
* fx-team: 11
* try: 5
* mozilla-aurora: 4
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux64: 28
* linux32: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-02-22&endday=2016-02-28&tree=all

Comment 25

[Intermittent Failures Robot]

54 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 31
* fx-team: 11
* try: 4
* mozilla-central: 3
* mozilla-aurora: 3
* mozilla-esr45: 2

Platform breakdown:
* linux64: 40
* linux32: 12
* osx-10-10: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-02-29&endday=2016-03-06&tree=all

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Bill, this is pretty frequent on Linux e10s. Any chance you can take a look please?

Comment 27

[Intermittent Failures Robot]

51 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 28
* fx-team: 10
* try: 6
* mozilla-central: 3
* mozilla-beta: 2
* mozilla-aurora: 2

Platform breakdown:
* linux64: 36
* linux32: 13
* windowsxp: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-03-07&endday=2016-03-13&tree=all

Comment 28

[Intermittent Failures Robot]

35 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 21
* mozilla-beta: 4
* fx-team: 4
* mozilla-aurora: 3
* try: 2
* ash: 1

Platform breakdown:
* linux64: 23
* linux32: 11
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-03-14&endday=2016-03-20&tree=all

Comment 29

[Brad Lassey [:blassey] (use needinfo?)]

Kyle, can you try to catch this in rr chaos mode?

Comment 30

[Intermittent Failures Robot]

38 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 19
* try: 6
* fx-team: 6
* mozilla-beta: 3
* mozilla-aurora: 2
* ash: 2

Platform breakdown:
* linux64: 22
* linux32: 15
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-03-21&endday=2016-03-27&tree=all

Comment 31

[Intermittent Failures Robot]

21 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* mozilla-beta: 5
* fx-team: 5
* mozilla-aurora: 3
* try: 2

Platform breakdown:
* linux64: 14
* linux32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-03-28&endday=2016-04-03&tree=all

Comment 32

[Intermittent Failures Robot]

25 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 8
* mozilla-inbound: 6
* mozilla-aurora: 4
* fx-team: 3
* ash: 2
* mozilla-central: 1
* mozilla-beta: 1

Platform breakdown:
* linux32: 11
* linux64: 8
* osx-10-10: 3
* windows8-64: 2
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-04-04&endday=2016-04-10&tree=all

Comment 33

[Ryan VanderMeulen [:RyanVM]]

Maybe third time will be the charm. Jed, any chance you could take a look? :)

Comment 34

[Ryan VanderMeulen [:RyanVM]]

For anybody looking into this, it mostly happens on Linux (32 and 64, mostly debug but with an occasional opt hit as well), but there have been a few cases on OSX and Windows as well. All the logs I've looked at show this as happening after running through dom/base/test/ directory.

Comment 35

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

So we ActorDestroy a BlobChild, and it calls NoteDyingActor before doing the Destroy() stuff so that it doesn't get Send__delete__()d by its RemoteBlobImpl when it's already destroyed, and that's good… but its mBlobImpl is a RemoteBlobSliceImpl, which holds the last reference to another RemoteBlobImpl (its mParent), whose mActor *hasn't* been zapped yet.  I think.

This reminds me of bug 1202887 comment #2, but in that case the spurious Send__delete__ didn't assert (it was just subtree destruction, not closing the entire transport), and instead caused a use-after-free later on.  So that's a little different, but I think it's the same genre of footgun.

I'm not sure what the right fix here is.  If this is always during shutdown, detecting that and just skipping the Send__delete__ might be enough.  Otherwise, there's always deferring it via NS_DispatchToCurrentThread like I did with the nsWebBrowserPersist thing (which will fail during shutdown, I think? In which case that falls back to the previous item).

Comment 36

[Ryan VanderMeulen [:RyanVM]]

I managed to bisect this on inbound down to bug 1225202 (TIL: Treeherder's "Add New Jobs" functionality even works for pushes from 5 months ago!). A Try push with test_fileapi_slice.html disabled confirms that the crashes go away.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a3cab87ef3143845bf9a65d18969f026c0cd6cb3

Maybe another round of rr chaos mode is in order now that we know which test to target?

Comment 37

[Intermittent Failures Robot]

36 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 16
* mozilla-inbound: 8
* fx-team: 6
* mozilla-aurora: 3
* mozilla-central: 1
* mozilla-beta: 1
* ash: 1

Platform breakdown:
* linux64: 19
* linux32: 11
* windows7-32: 4
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-04-11&endday=2016-04-17&tree=all

Comment 38

[Intermittent Failures Robot]

23 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 11
* mozilla-aurora: 5
* fx-team: 4
* mozilla-release: 1
* mozilla-beta: 1
* ash: 1

Platform breakdown:
* linux64: 11
* linux32: 11
* windowsxp: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-04-18&endday=2016-04-24&tree=all

Comment 39

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/905df9d5ab74

Comment 40

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/905df9d5ab74

Comment 41

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/2545c54d540ee002c2366ed01e4233d98614298d

Comment 42

[Intermittent Failures Robot]

7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* try: 4
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* linux64: 3
* windowsxp: 2
* windows7-32: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-04-25&endday=2016-05-01&tree=all

Comment 43

[Ryan VanderMeulen [:RyanVM]]

Try indicates that whatever was broken here appears to have gone away on its own.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c85d2ecf2d1a

I'll go ahead and re-enable it and we can always re-open the bug and re-disable if Try is lying.

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c673ca80496
Re-enable test_fileapi_slice.html on e10s.

Comment 45

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6c673ca80496

Comment 46

[Intermittent Failures Robot]

8 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 3
* autoland: 3
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* osx-10-10: 3
* windows7-32-vm: 2
* linux64: 2
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1226477&startday=2016-10-03&endday=2016-10-09&tree=all