Closed Bug 1348682 Opened 7 years ago Closed 5 years ago

Intermittent xpcshell-unpack.ini:toolkit/mozapps/extensions/test/xpcshell/test_bug542391.js | application crashed [@ mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::OpenDirectory] after Assertion failure: !QuotaClient::IsShuttingDownOnBackgroun

Categories

(Core :: Storage: IndexedDB, defect)

Product:
Core
Component:
Storage: IndexedDB
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla71
Tracking Flags:
Tracking Status
firefox-esr60 --- wontfix
firefox-esr68 --- fixed
firefox69 --- wontfix
firefox70 --- fixed
firefox71 --- fixed

People

(Reporter: aryx, Assigned: tt)

References

Details

(Keywords: assertion, intermittent-failure, regression, Whiteboard: [stockwell disable-recommended])

Attachments

(1 file)

Bug 1348682 - Check if it's shutting down at the beginning of the State::FinishOpen;
47 bytes, text/x-phabricator-request
lizzard
: approval-mozilla-beta+
lizzard
: approval-mozilla-esr68+
Details | Review 
https://treeherder.mozilla.org/logviewer.html#?job_id=84895628&repo=autoland

08:28:26     INFO -  PID 10992 | Assertion failure: !QuotaClient::IsShuttingDownOnBackgroundThread(), at c:/builds/moz2_slave/autoland-w64-d-000000000000000/build/src/dom/indexedDB/ActorsParent.cpp:21408
08:28:37     INFO -  PID 10992 | #01: mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::FinishOpen [dom/indexedDB/ActorsParent.cpp:21369]
08:28:37     INFO -  PID 10992 | #02: mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::Run [dom/indexedDB/ActorsParent.cpp:21522]
08:28:37     INFO -  PID 10992 | #03: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:1270]
08:28:37     INFO -  PID 10992 | #04: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/threads/nsThreadUtils.cpp:389]
08:28:37     INFO -  PID 10992 | #05: mozilla::dom::indexedDB::`anonymous namespace'::ConnectionPool::Shutdown [dom/indexedDB/ActorsParent.cpp:12607]
08:28:37     INFO -  PID 10992 | #06: mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::ShutdownWorkThreads [dom/indexedDB/ActorsParent.cpp:18203]
08:28:37     INFO -  PID 10992 | #07: mozilla::dom::quota::QuotaManager::Shutdown() [dom/quota/ActorsParent.cpp:3389]
08:28:37     INFO -  PID 10992 | #08: mozilla::dom::quota::QuotaManager::ShutdownRunnable::Run() [dom/quota/ActorsParent.cpp:2601]
08:28:37     INFO -  PID 10992 | #09: nsThread::ProcessNextEvent(bool,bool *) [xpcom/threads/nsThread.cpp:1270]
08:28:37     INFO -  PID 10992 | #10: NS_ProcessNextEvent(nsIThread *,bool) [xpcom/threads/nsThreadUtils.cpp:389]
08:28:37     INFO -  PID 10992 | #11: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate *) [ipc/glue/MessagePump.cpp:368]
08:28:37     INFO -  PID 10992 | #12: MessageLoop::RunHandler() [ipc/chromium/src/base/message_loop.cc:232]
08:28:37     INFO -  PID 10992 | #13: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:212]
08:28:37     INFO -  PID 10992 | #14: nsThread::ThreadFunc(void *) [xpcom/threads/nsThread.cpp:502]
08:28:37     INFO -  PID 10992 | #15: PR_NativeRunThread [nsprpub/pr/src/threads/combined/pruthr.c:406]
08:28:37     INFO -  PID 10992 | #16: pr_root [nsprpub/pr/src/md/windows/w95thred.c:96]
08:28:37     INFO -  PID 10992 | #17: ucrtbase.DLL + 0x1cab0
08:28:37     INFO -  PID 10992 | #18: KERNEL32.DLL + 0x167e
08:28:37     INFO -  PID 10992 | #19: ntdll.dll + 0x1c3f1
08:28:37     INFO -  <<<<<<<
08:28:37     INFO -  mozcrash Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld~1.t-w\appdata\local\temp\xpc-other-qjyi53\3c6506ce-c0f6-4bf8-94d1-cd6337c253df.dmp C:\slave\test\build\symbols
08:28:43     INFO -  mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\3c6506ce-c0f6-4bf8-94d1-cd6337c253df.dmp
08:28:43     INFO -  mozcrash Saved app info as C:\slave\test\build\blobber_upload_dir\3c6506ce-c0f6-4bf8-94d1-cd6337c253df.extra
08:28:43  WARNING -  PROCESS-CRASH | xpcshell-unpack.ini:toolkit/mozapps/extensions/test/xpcshell/test_bug542391.js | application crashed [@ mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::OpenDirectory]
08:28:43     INFO -  Crash dump filename: c:\users\cltbld~1.t-w\appdata\local\temp\xpc-other-qjyi53\3c6506ce-c0f6-4bf8-94d1-cd6337c253df.dmp
08:28:43     INFO -  Operating system: Windows NT
08:28:43     INFO -                    6.2.9200
08:28:43     INFO -  CPU: amd64
08:28:43     INFO -       family 6 model 30 stepping 5
08:28:43     INFO -       8 CPUs
08:28:43     INFO -  GPU: UNKNOWN
08:28:43     INFO -  Crash reason:  EXCEPTION_BREAKPOINT
08:28:43     INFO -  Crash address: 0x7fc6d74b0b6
08:28:43     INFO -  Process uptime: 6 seconds
08:28:43     INFO -  Thread 24 (crashed)
08:28:43     INFO -   0  xul.dll!mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::OpenDirectory [ActorsParent.cpp:5f099873b15d : 21408 + 0x1b]
08:28:43     INFO -      rax = 0x0000000000000000   rdx = 0x00000016c58ff448
08:28:43     INFO -      rcx = 0x00000000ffffffff   rbx = 0x00000000000053a0
08:28:43     INFO -      rsi = 0x0000000000000000   rdi = 0x00000016c562b4c0
08:28:43     INFO -      rbp = 0x00000016c58ff530   rsp = 0x00000016c58ff430
08:28:43     INFO -       r8 = 0x00000016c58ff440    r9 = 0x00000016c58ff438
08:28:43     INFO -      r10 = 0x0000000000000000   r11 = 0x00000016c58fa980
08:28:43     INFO -      r12 = 0x0000000000000000   r13 = 0x0000000000000000
08:28:43     INFO -      r14 = 0x00000016b9a5be01   r15 = 0x0000000000000000
08:28:43     INFO -      rip = 0x000007fc6d74b0b6
08:28:43     INFO -      Found by: given as instruction pointer in context
08:28:43     INFO -   1  xul.dll!mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::FinishOpen [ActorsParent.cpp:5f099873b15d : 21368 + 0xd]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff5a0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6d73558f
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   2  xul.dll!mozilla::dom::indexedDB::`anonymous namespace'::FactoryOp::Run [ActorsParent.cpp:5f099873b15d : 21521 + 0x5]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff5e0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6d759866
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   3  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:5f099873b15d : 1269 + 0x14]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff620   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b30fc27
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   4  xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:5f099873b15d : 389 + 0xd]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff780   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b30ecf7
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   5  xul.dll!mozilla::dom::indexedDB::`anonymous namespace'::ConnectionPool::Shutdown [ActorsParent.cpp:5f099873b15d : 12607 + 0xa]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff7c0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6d761232
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   6  xul.dll!mozilla::dom::indexedDB::`anonymous namespace'::QuotaClient::ShutdownWorkThreads [ActorsParent.cpp:5f099873b15d : 18201 + 0x8]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff830   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6d761784
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   7  xul.dll!mozilla::dom::quota::QuotaManager::Shutdown() [ActorsParent.cpp:5f099873b15d : 3389 + 0x26]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff860   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6d5dcf21
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   8  xul.dll!mozilla::dom::quota::QuotaManager::ShutdownRunnable::Run() [ActorsParent.cpp:5f099873b15d : 2599 + 0x8]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff8c0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6d5dbfc7
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -   9  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:5f099873b15d : 1269 + 0x14]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ff8f0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b30fc27
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  10  xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:5f099873b15d : 389 + 0xd]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffa50   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b30ecf7
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  11  xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate *) [MessagePump.cpp:5f099873b15d : 368 + 0xa]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffa90   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b81b953
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  12  xul.dll!MessageLoop::RunHandler() [message_loop.cc:5f099873b15d : 231 + 0x5]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffaf0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b7e05ce
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  13  xul.dll!MessageLoop::Run() [message_loop.cc:5f099873b15d : 211 + 0x8]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffb20   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b7e0292
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  14  xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:5f099873b15d : 500 + 0x32]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffb70   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc6b31480c
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  15  nss3.dll!PR_NativeRunThread [pruthr.c:5f099873b15d : 397 + 0x7]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffbd0   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc7306255a
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  16  nss3.dll!pr_root [w95thred.c:5f099873b15d : 95 + 0x6]
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffc00   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc73053d6e
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  17  ucrtbase.dll!o__realloc_base + 0x60
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffc30   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc739acab0
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  18  kernel32.dll!BaseThreadInitThunk + 0x1a
08:28:43     INFO -      rbx = 0x00000000000053a0   rbp = 0x00000016c58ff530
08:28:43     INFO -      rsp = 0x00000016c58ffc60   r12 = 0x0000000000000000
08:28:43     INFO -      r13 = 0x0000000000000000   r14 = 0x00000016b9a5be01
08:28:43     INFO -      r15 = 0x0000000000000000   rip = 0x000007fc8bae167e
08:28:43     INFO -      Found by: call frame info
08:28:43     INFO -  19  ntdll.dll!RtlUserThreadStart + 0x21
08:28:43     INFO -      rsp = 0x00000016c58ffc90   rip = 0x000007fc8bc3c3f1
08:28:43     INFO -      Found by: stack scanning
08:28:43     INFO -  20  KERNELBASE.dll!GetLegacyComposition + 0x1180
08:28:43     INFO -      rsp = 0x00000016c58ffcc0   rip = 0x000007fc88dd09d0
08:28:43     INFO -      Found by: stack scanning
Priority: -- → P5

Hi Nicolas, the failure rate on this bug started increasing with the landing of bug 1576085, but on different tests not specifically xpcshell.

https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=263490137&revision=8098c67fb538ba30483418e04ee5f1554a65eefb

Can you please take a look?

Flags: needinfo?(nchevobbe)
Whiteboard: [stockwell needswork:owner]

Bug 1576085 only impacts the WebConsole client codebase, which looks totally unrelated to the test failures.

Baku, I see you reviewed a patch modifying the area of the code that has this assertions. Would you know in which case we might hit it?

For reference, here's the error and its stacktrace:

Assertion failure: !QuotaClient::IsShuttingDownOnBackgroundThread(), at /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp:19721

#01: mozilla::dom::indexedDB::(anonymous namespace)::FactoryOp::Run() [dom/indexedDB/ActorsParent.cpp:19828]
#02: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1214]
#03: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:486]
#04: mozilla::dom::indexedDB::(anonymous namespace)::QuotaClient::ShutdownWorkThreads() [dom/indexedDB/ActorsParent.cpp:16278]
#05: mozilla::dom::quota::QuotaManager::Shutdown() [dom/quota/ActorsParent.cpp:3815]
#06: mozilla::dom::quota::QuotaManager::ShutdownInstance() [dom/quota/ActorsParent.cpp:3405]
#07: mozilla::dom::quota::RecvShutdownQuotaManager() [dom/quota/ActorsParent.cpp:2663]
#08: mozilla::ipc::BackgroundParentImpl::RecvShutdownQuotaManager() [ipc/glue/BackgroundParentImpl.cpp:1037]
#09: mozilla::ipc::PBackgroundParent::OnMessageReceived(IPC::Message const&) [s3:gecko-generated-sources:20735819896cce1c0f622ea88c03b28874224da5f137fcdf0288698335f47f38344cb74c308d040c16d76f70baf5bacb2d5026c86885513268c9c47b486d13c0/ipc/ipdl/PBackgroundParent.cpp::0]
#10: mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) [ipc/glue/MessageChannel.cpp:2185]
#11: mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) [ipc/glue/MessageChannel.cpp:2108]
#12: mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) [ipc/glue/MessageChannel.cpp:0]
#13: mozilla::ipc::MessageChannel::MessageTask::Run() [ipc/glue/MessageChannel.cpp:1988]
#14: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1214]
#15: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:486]
#16: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [ipc/glue/MessagePump.cpp:301]
#17: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:291]
#18: nsThread::ThreadFunc(void*) [xpcom/threads/nsThread.cpp:460]
#19: _pt_root [nsprpub/pr/src/pthreads/ptthread.c:201]
#20: libsystem_pthread.dylib + 0x32eb
#21: libsystem_pthread.dylib + 0x6249
Flags: needinfo?(nchevobbe) → needinfo?(amarchesini)
Flags: needinfo?(amarchesini) → needinfo?(bugmail)
Flags: needinfo?(bugmail) → needinfo?(jvarga)
Flags: needinfo?(jvarga) → needinfo?(shes050117)

There are 83 total failures in the last 7 days, with a spike starting from the 25th of August, on:

*debug: linux64, linux64-ccov, linux64-qr, macosx1014-64, macosx1014-64-qr, windows7-32

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=263994182&repo=mozilla-central&lineNumber=68696

[task 2019-08-29T05:56:47.468Z] 05:56:47 INFO - TEST-START | devtools/server/tests/browser/browser_accessibility_infobar_audit_text_label.js
[task 2019-08-29T05:56:47.480Z] 05:56:47 INFO - GECKO(3258) | ++DOCSHELL 0x11b3bb000 == 5 [pid = 3263] [id = {ca711fe7-9e75-5244-aa77-97778e676975}]
[task 2019-08-29T05:56:47.480Z] 05:56:47 INFO - GECKO(3258) | ++DOMWINDOW == 12 (0x102fd6f20) [pid = 3263] [serial = 12] [outer = 0x0]
[task 2019-08-29T05:56:47.480Z] 05:56:47 INFO - GECKO(3258) | ++DOMWINDOW == 13 (0x11d82b800) [pid = 3263] [serial = 13] [outer = 0x102fd6f20]
[task 2019-08-29T05:56:47.684Z] 05:56:47 INFO - GECKO(3258) | ++DOMWINDOW == 14 (0x11cf34400) [pid = 3263] [serial = 14] [outer = 0x102fd6f20]
[task 2019-08-29T05:56:47.930Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x1257d0800 == 5 [pid = 3261] [id = {ac2a37a9-272d-744c-8126-8d9b9b2bbf53}] [url = about:blank]
[task 2019-08-29T05:56:47.930Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x1262da800 == 4 [pid = 3261] [id = {37a9096c-86d8-124a-99d4-9af00a28c0b8}] [url = about:blank]
[task 2019-08-29T05:56:47.930Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x1257cf800 == 3 [pid = 3261] [id = {19078ac1-b30a-a349-85af-a34884e6cf67}] [url = http://test1.example.org/browser/devtools/server/tests/browser/doc_accessibility_infobar.html]
[task 2019-08-29T05:56:47.931Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x11fbed000 == 2 [pid = 3261] [id = {4d5021b6-edf6-4a44-9b68-94cc749e4a4e}] [url = about:blank]
[task 2019-08-29T05:56:47.931Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x11fbe4800 == 1 [pid = 3261] [id = {30e3fc67-cd6d-c542-bdcd-cbf67863ff1a}] [url = about:blank]
[task 2019-08-29T05:56:47.931Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x11fbeb000 == 0 [pid = 3261] [id = {3082e874-83d7-5b46-ae76-7c545c507840}] [url = http://test1.example.org/browser/devtools/server/tests/browser/doc_accessibility_infobar.html]
[task 2019-08-29T05:56:47.997Z] 05:56:47 INFO - GECKO(3258) | --DOCSHELL 0x119e09800 == 0 [pid = 3264] [id = {cf51af51-50fb-f848-b712-80d2d55ac9d2}] [url = http://test1.example.org/browser/devtools/server/tests/browser/doc_accessibility_infobar.html]
[task 2019-08-29T05:56:48.181Z] 05:56:48 INFO - GECKO(3258) | JavaScript error: resource://testing-common/PromiseTestUtils.jsm, line 112: uncaught exception: Object
[task 2019-08-29T05:56:48.181Z] 05:56:48 INFO - GECKO(3258) | MEMORY STAT | vsize 7615MB | residentFast 359MB | heapAllocated 148MB
[task 2019-08-29T05:56:48.182Z] 05:56:48 INFO - TEST-OK | devtools/server/tests/browser/browser_accessibility_infobar_audit_text_label.js | took 709ms

[task 2019-08-29T05:56:56.828Z] 05:56:56 INFO - GECKO(3258) | Assertion failure: !QuotaClient::IsShuttingDownOnBackgroundThread(), at /builds/worker/workspace/build/src/dom/indexedDB/ActorsParent.cpp:19721
[task 2019-08-29T05:57:16.407Z] 05:57:16 INFO - GECKO(3258) | #01: mozilla::dom::indexedDB::(anonymous namespace)::FactoryOp::Run() [dom/indexedDB/ActorsParent.cpp:19828]
[task 2019-08-29T05:57:16.407Z] 05:57:16 INFO -
[task 2019-08-29T05:57:16.407Z] 05:57:16 INFO - GECKO(3258) | #02: nsThread::ProcessNextEvent(bool, bool*) [xpcom/threads/nsThread.cpp:1214]
[task 2019-08-29T05:57:16.407Z] 05:57:16 INFO -
[task 2019-08-29T05:57:16.408Z] 05:57:16 INFO - GECKO(3258) | #03: NS_ProcessNextEvent(nsIThread*, bool) [xpcom/threads/nsThreadUtils.cpp:486]
[task 2019-08-29T05:57:16.408Z] 05:57:16 INFO -
[task 2019-08-29T05:57:16.408Z] 05:57:16 INFO - GECKO(3258) | #04: mozilla::dom::indexedDB::(anonymous namespace)::QuotaClient::ShutdownWorkThreads() [dom/indexedDB/ActorsParent.cpp:16278]
[task 2019-08-29T05:57:16.408Z] 05:57:16 INFO -
[task 2019-08-29T05:57:16.408Z] 05:57:16 INFO - GECKO(3258) | #05: mozilla::dom::quota::QuotaManager::Shutdown() [dom/quota/ActorsParent.cpp:3815]

This will soon reach the disable-recommended queue, however since the assertion occurs in different tests we cannot disable it. Please take a look.

So, before jumping into the crashing state on the Background thread, it was on either [1] or [2] (two of cases are on the main thread). And, we crash in [3] because it asserts that we should be in shutting down.

To me, it seems it's possible that QM/IDB receives the shutting down notification between [1, 2] and [3]. So, I'm going to provide a patch to check if it's shutting down at the beginning of the State::FinishOpen [4].

[1] FactoryOp::Open(), https://searchfox.org/mozilla-central/rev/e8f334670ddf4879a80be7af79651c06592e1c3e/dom/indexedDB/ActorsParent.cpp#19225-19229
[2] FactoryOp::RetryCheckPermission(), https://searchfox.org/mozilla-central/rev/e8f334670ddf4879a80be7af79651c06592e1c3e/dom/indexedDB/ActorsParent.cpp#19322-19326
[3] https://searchfox.org/mozilla-central/rev/e8f334670ddf4879a80be7af79651c06592e1c3e/dom/indexedDB/ActorsParent.cpp#19721
[4] https://searchfox.org/mozilla-central/rev/e8f334670ddf4879a80be7af79651c06592e1c3e/dom/indexedDB/ActorsParent.cpp#19679

Assignee: nobody → shes050117
Flags: needinfo?(shes050117)
Pushed by rmaries@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/213df6004244
Check if it's shutting down at the beginning of the State::FinishOpen; r=asuth

https://hg.mozilla.org/mozilla-central/rev/213df6004244

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox71: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla71

This is still occurring on mozilla-beta, can you request uplift?

Flags: needinfo?(shes050117)

(In reply to Andreea Pavel [:apavel] from comment #42)

This is still occurring on mozilla-beta, can you request uplift?

ESR68 too please.

status-firefox69: --- → wontfix
status-firefox70: --- → affected
status-firefox-esr60: --- → wontfix
status-firefox-esr68: --- → affected

Comment on attachment 9089696 [details]
Bug 1348682 - Check if it's shutting down at the beginning of the State::FinishOpen;

Beta/Release Uplift Approval Request

  • User impact if declined: This patch checks if it's shutting down at the beginning of Finish::Open state. If it is declined, then it might cause a delay of shutting down and something unexpected since there is an assertion to assume we cannot shut down on the code a few lines below.

The code is not covered by the test directly, but the failure rate of some tests (for crashing) has dropped.

  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): It's just an if-check at the beginning of the stage, so the risk should be low.
  • String changes made/needed:
Flags: needinfo?(shes050117)
Attachment #9089696 - Flags: approval-mozilla-beta?

Comment on attachment 9089696 [details]
Bug 1348682 - Check if it's shutting down at the beginning of the State::FinishOpen;

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: The fix has been verified on Nightly, but ESR68 still has similar failures.
  • User impact if declined: If it is declined, then it might cause a delay of shutting down and something unexpected since there is an assertion to assume we cannot shut down on the code a few lines below.
  • Fix Landed on Version: 71
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): It's just an if-check at the beginning of the stage, so the risk should be low.
  • String or UUID changes made by this patch:
Attachment #9089696 - Flags: approval-mozilla-esr68?

Comment on attachment 9089696 [details]
Bug 1348682 - Check if it's shutting down at the beginning of the State::FinishOpen;

Fix for shutdown problems, OK for uplift for beta 6/esr

Attachment #9089696 - Flags: approval-mozilla-esr68?
Attachment #9089696 - Flags: approval-mozilla-esr68+
Attachment #9089696 - Flags: approval-mozilla-beta?
Attachment #9089696 - Flags: approval-mozilla-beta+

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression

(In reply to Intermittent Failures Robot from comment #56)

1 failures in 4314 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

  • autoland: 1

Platform breakdown:

  • linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1348682&startday=2019-10-28&endday=2019-11-03&tree=all

At least this last one is a misclassification. The assertion failure here is:

Assertion failure: !IsShuttingDown(), at /builds/worker/workspace/build/src/dom/ipc/ContentChild.cpp:1873
Flags: needinfo?(aryx.bugmail)

(In reply to Intermittent Failures Robot from comment #59)

1 failures in 5582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

  • mozilla-central: 1

Platform breakdown:

  • linux1804-64-asan: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1348682&startday=2020-04-20&endday=2020-04-26&tree=all

This occurrence seems unrelated, too:

#1 0x7f8e3fcf90cf in mozilla::AppWindow::GetPrimaryRemoteTabSize(int*, int*) /builds/worker/checkouts/gecko/xpfe/appshell/AppWindow.cpp:2003:34
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: