Closed Bug 1558103 Opened 5 years ago Closed 3 years ago

Intermittent test/xpcshell/<test-name> | application crashed [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)]

Categories

(GeckoView :: General, defect, P1)

Tracking

(firefox88 wontfix, firefox89 wontfix, firefox90 fixed)

RESOLVED FIXED
90 Branch
Tracking Status
firefox88 --- wontfix
firefox89 --- wontfix
firefox90 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: agi)

References

(Regression)

Details

(Keywords: crash, intermittent-failure, regression, Whiteboard: [geckoview:m90][stockwell unknown])

Crash Data

Attachments

(1 file)

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=250861842&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/PehE7kGLSRKX6angey9RRA/runs/0/artifacts/public/logs/live_backing.log


00:25:11 INFO - TEST-START | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_proxy_authorization_via_proxyinfo.js
00:25:11 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_incognito.js | took 5160ms

00:25:15 WARNING - PROCESS-CRASH | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_background_runtime_connect_params.js | application crashed [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)]
00:25:15 INFO - Crash dump filename: c:\users\task_1560122237\appdata\local\temp\xpc-other-wmhou8\c42a98e2-0c76-4e34-a1b8-29fdf6e76acd.dmp
00:25:15 INFO - Operating system: Windows NT
00:25:15 INFO - 6.1.7601 Service Pack 1
00:25:15 INFO - CPU: x86
00:25:15 INFO - GenuineIntel family 6 model 63 stepping 2
00:25:15 INFO - 8 CPUs
00:25:15 INFO - GPU: UNKNOWN
00:25:15 INFO - Crash reason: EXCEPTION_BREAKPOINT
00:25:15 INFO - Crash address: 0x6a4a8041
00:25:15 INFO - Assertion: Unknown assertion type 0x00000000
00:25:15 INFO - Process uptime: 2 seconds
00:25:15 INFO - Thread 16 (crashed)
00:25:15 INFO - 0 mozglue.dll!mozalloc_abort [mozalloc_abort.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 33 + 0x0]
00:25:15 INFO - eip = 0x6a4a8041 esp = 0x05ebed38 ebp = 0x05ebed40 ebx = 0x05ebed78
00:25:15 INFO - esi = 0x05ebed78 edi = 0x6a22f9b0 eax = 0x00000000 ecx = 0x00000000
00:25:15 INFO - edx = 0x6a30e340 efl = 0x00000206
00:25:15 INFO - Found by: given as instruction pointer in context
00:25:15 INFO - 1 xul.dll!static void Abort(const char *) [nsDebugImpl.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 439 + 0xa]
00:25:15 INFO - eip = 0x54da97aa esp = 0x05ebed48 ebp = 0x05ebed4c esi = 0x6a22f9b0
00:25:15 INFO - edi = 0x00552f82
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 2 xul.dll!NS_DebugBreak [nsDebugImpl.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 429 + 0xb]
00:25:15 INFO - eip = 0x54da93e9 esp = 0x05ebed54 ebp = 0x05ebf180
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 3 xul.dll!mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int) [ThreadEventTarget.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 117 + 0x18]
00:25:15 INFO - eip = 0x54e7ad84 esp = 0x05ebf188 ebp = 0x05ebf1bc ebx = 0x009014f0
00:25:15 INFO - esi = 0x05ebf1c4 edi = 0x009027e0
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 4 xul.dll!nsThread::Dispatch(already_AddRefed<nsIRunnable>,unsigned int) [nsThread.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 712 + 0xa]
00:25:15 INFO - eip = 0x54e84fd3 esp = 0x05ebf1c4 ebp = 0x05ebf1e0 ebx = 0x05ebf1d0
00:25:15 INFO - esi = 0x00000000 edi = 0x05ebf1e8
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 5 xul.dll!mozilla::ProcessHangMonitor::Dispatch(already_AddRefed<nsIRunnable>) [ProcessHangMonitor.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 1344 + 0xe]
00:25:15 INFO - eip = 0x58006afb esp = 0x05ebf1e8 ebp = 0x05ebf200 ebx = 0x05ebf220
00:25:15 INFO - esi = 0x05ebf1f4 edi = 0x05ebf208
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 6 xul.dll!void anonymous namespace'::HangMonitorChild::Dispatch(struct already_AddRefed<nsIRunnable>) [ProcessHangMonitor.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 132 + 0x5] 00:25:15 INFO - eip = 0x58007b81 esp = 0x05ebf208 ebp = 0x05ebf218 esi = 0x05ebf20c 00:25:15 INFO - edi = 0x05ebf220 00:25:15 INFO - Found by: call frame info 00:25:15 INFO - 7 xul.dll!voidanonymous namespace'::HangMonitorChild::ActorDestroy(mozilla::ipc::IProtocol::ActorDestroyReason) [ProcessHangMonitor.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 496 + 0x5]
00:25:15 INFO - eip = 0x580080a2 esp = 0x05ebf220 ebp = 0x05ebf238 esi = 0x0095d400
00:25:15 INFO - edi = 0x009014f0
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 8 xul.dll!mozilla::ipc::IProtocol::DestroySubtree(mozilla::ipc::IProtocol::ActorDestroyReason) [ProtocolUtils.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 659 + 0xa]
00:25:15 INFO - eip = 0x5554e75b esp = 0x05ebf240 ebp = 0x05ebf25c ebx = 0x00000000
00:25:15 INFO - esi = 0x0095d400 edi = 0x00000004
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 9 xul.dll!mozilla::PProcessHangMonitorChild::OnChannelError() [PProcessHangMonitorChild.cpp: : 369 + 0x7]
00:25:15 INFO - eip = 0x5584898d esp = 0x05ebf264 ebp = 0x05ebf26c ebx = 0x00000000
00:25:15 INFO - esi = 0x0095d400 edi = 0x00905300
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 10 xul.dll!nsresult mozilla::detail::RunnableMethodImpl<mozilla::ipc::MessageChannel ,void (mozilla::ipc::MessageChannel::)() attribute((thiscall)),0,mozilla::RunnableKind::Cancelable>::Run() [nsThreadUtils.h:7bea9b04f35428cc32dc266d7a589f138a063137 : 1176 + 0x6]
00:25:15 INFO - eip = 0x555542c3 esp = 0x05ebf274 ebp = 0x05ebf274 esi = 0x05ebf280
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 11 xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 1176 + 0x6]
00:25:15 INFO - eip = 0x54e86c5e esp = 0x05ebf27c ebp = 0x05ebf74c
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 12 xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 486 + 0x11]
00:25:15 INFO - eip = 0x54e89e44 esp = 0x05ebf754 ebp = 0x05ebf770 ebx = 0x08452100
00:25:15 INFO - esi = 0x05ebf763 edi = 0x0098d900
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 13 xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate *) [MessagePump.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 333 + 0xa]
00:25:15 INFO - eip = 0x5554a16a esp = 0x05ebf778 ebp = 0x05ebf7a4 ebx = 0x08452100
00:25:15 INFO - esi = 0x08452140
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 14 xul.dll!MessageLoop::RunInternal() [message_loop.cc:7bea9b04f35428cc32dc266d7a589f138a063137 : 315 + 0x5]
00:25:15 INFO - eip = 0x554ff2a1 esp = 0x05ebf7ac ebp = 0x05ebf7c8 ebx = 0x00901550
00:25:15 INFO - esi = 0x0098d900 edi = 0x0098d900
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 15 xul.dll!MessageLoop::RunHandler() [message_loop.cc:7bea9b04f35428cc32dc266d7a589f138a063137 : 308 + 0x7]
00:25:15 INFO - eip = 0x554ff1bc esp = 0x05ebf7d0 ebp = 0x05ebf7fc esi = 0x0098d900
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 16 xul.dll!MessageLoop::Run() [message_loop.cc:7bea9b04f35428cc32dc266d7a589f138a063137 : 290 + 0x5]
00:25:15 INFO - eip = 0x554ff08f esp = 0x05ebf804 ebp = 0x05ebf81c ebx = 0x00901550
00:25:15 INFO - esi = 0x00905300 edi = 0x0098d900
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 17 xul.dll!nsThread::ThreadFunc(void *) [nsThread.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 455 + 0xa]
00:25:15 INFO - eip = 0x54e8397b esp = 0x05ebf824 ebp = 0x05ebf848
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 18 nss3.dll!_PR_NativeRunThread [pruthr.c:7bea9b04f35428cc32dc266d7a589f138a063137 : 397 + 0x6]
00:25:15 INFO - eip = 0x6a052b28 esp = 0x05ebf850 ebp = 0x05ebf870 ebx = 0x6a23e130
00:25:15 INFO - esi = 0x08408e10 edi = 0x08408d60
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 19 nss3.dll!static unsigned int pr_root(void *) [w95thred.c:7bea9b04f35428cc32dc266d7a589f138a063137 : 137 + 0x7]
00:25:15 INFO - eip = 0x6a04155d esp = 0x05ebf878 ebp = 0x05ebf87c ebx = 0x6a23e130
00:25:15 INFO - esi = 0x6a041550 edi = 0x00151e28
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 20 ucrtbase.dll!__RtlConvertUlongToLargeInteger + 0x8489
00:25:15 INFO - eip = 0x6a23e16f esp = 0x05ebf884 ebp = 0x05ebf8b8
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 21 kernel32.dll! ?? ?? ::FNODOBFM::`string' + 0x46
00:25:15 INFO - eip = 0x75b0ef3c esp = 0x05ebf8c0 ebp = 0x05ebf8c4
00:25:15 INFO - Found by: previous frame's frame pointer
00:25:15 INFO - 22 mozglue.dll!static void patched_BaseThreadInitThunk(int, void *, void *) [WindowsDllBlocklist.cpp:7bea9b04f35428cc32dc266d7a589f138a063137 : 693 + 0x41]
00:25:15 INFO - eip = 0x6a486fc4 esp = 0x05ebf8cc ebp = 0x05ebf8fc
00:25:15 INFO - Found by: previous frame's frame pointer
00:25:15 INFO - 23 ntdll.dll!__RtlUserThreadStart + 0x27
00:25:15 INFO - eip = 0x771b3618 esp = 0x05ebf904 ebp = 0x05ebf93c ebx = 0x00151e28
00:25:15 INFO - esi = 0x00000000 edi = 0x00000000
00:25:15 INFO - Found by: call frame info
00:25:15 INFO - 24 ntdll.dll!_RtlUserThreadStart + 0x1b
00:25:15 INFO - eip = 0x771b35eb esp = 0x05ebf944 ebp = 0x05ebf954
00:25:15 INFO - Found by: call frame info

Component: General → DOM: Content Processes
Product: WebExtensions → Core
Summary: Intermittent toolkit/components/extensions/test/xpcshell/test_ext_background_runtime_connect_params.js | application crashed [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)] → Intermittent test/xpcshell/<test-name> | application crashed [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)]

The priority flag is not set for this bug.
:jimm, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(jmathies)
Flags: needinfo?(jmathies)
Priority: -- → P3
Crash Signature: [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)] → [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)] [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>, unsigned int)]

Update:

There have been 57 failures within the last 7 days, all of them on Android 7.0 x86-64 debug.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=336903183&repo=autoland&lineNumber=8226

:neha, as the owner of the component, could you help us assign this to someone?
Thank you.

Crash Signature: [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)] [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>, unsigned int)] → [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>,unsigned int)] [@ mozilla::ThreadEventTarget::Dispatch(already_AddRefed<nsIRunnable>, unsigned int)]
Flags: needinfo?(nkochar)
Whiteboard: [stockwell needswork:owner]

Agi, this intermittent xpcshell test failure looks like a recent Android-only regression starting on March 23 or 24:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-03-01&endday=2021-04-19&tree=trunk&bug=1558103

I suspect this is fallout from you enabling xpcshell tests in bug 1567341.

Looks like we are hitting NS_ASSERTION(false, "Failed Dispatch after xpcom-shutdown-threads") during shutdown here:

https://searchfox.org/mozilla-central/rev/46a67b8656ac12b5c180e47bc4055f713d73983b/xpcom/threads/ThreadEventTarget.cpp#65

Flags: needinfo?(nkochar) → needinfo?(agi)
Regressed by: 1567341

Yep, thanks.

Assignee: nobody → agi
Component: DOM: Content Processes → General
Flags: needinfo?(agi)
Product: Core → GeckoView
Severity: critical → --
Priority: P3 → --

This might be another variation of Bug 1700775, let's see.

Whiteboard: [stockwell needswork:owner] → [stockwell needswork:owner][geckoview:m90?]
Whiteboard: [stockwell needswork:owner][geckoview:m90?] → [stockwell needswork:owner][geckoview:m90]
Severity: -- → S2
Priority: -- → P1

Added some logging on try, looks like this is caused by us calling Binding.onServiceDisconnected while shutting down, which makes sense to me. Likely all we have to do here is checking whether we're shutting down and if so ignore the onServiceDisconnected call.

04-22 20:58:32.052  9018  9018 E sferrog : dispatchNative
04-22 20:58:32.052  9018  9018 E sferrog :       at org.mozilla.gecko.util.XPCOMEventTarget.execute(XPCOMEventTarget.java:24)
04-22 20:58:32.052  9018  9018 E sferrog :       at org.mozilla.gecko.util.XPCOMEventTarget.runOnLauncherThread(XPCOMEventTarget.java:56)
04-22 20:58:32.052  9018  9018 E sferrog :       at org.mozilla.gecko.process.ServiceAllocator$InstanceInfo$Binding.onServiceDisconnected(ServiceAllocator.java:95)
04-22 20:58:32.052  9018  9018 E sferrog :       at android.app.LoadedApk$ServiceDispatcher.doDeath(LoadedApk.java:1470)
04-22 20:58:32.052  9018  9018 E sferrog :       at android.app.LoadedApk$ServiceDispatcher$RunConnection.run(LoadedApk.java:1484)
04-22 20:58:32.052  9018  9018 E sferrog :       at android.os.Handler.handleCallback(Handler.java:751)
04-22 20:58:32.052  9018  9018 E sferrog :       at android.os.Handler.dispatchMessage(Handler.java:95)
04-22 20:58:32.052  9018  9018 E sferrog :       at android.os.Looper.loop(Looper.java:154)
04-22 20:58:32.052  9018  9018 E sferrog :       at android.app.ActivityThread.main(ActivityThread.java:6077)
04-22 20:58:32.052  9018  9018 E sferrog :       at java.lang.reflect.Method.invoke(Native Method)
04-22 20:58:32.052  9018  9018 E sferrog :       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
04-22 20:58:32.052  9018  9018 E sferrog :       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)
04-22 20:58:32.052  9018  9018 I Gecko   : [Parent 9018, Unnamed thread 7e6ebcb20cd0] ###!!! ASSERTION: Failed Dispatch after xpcom-shutdown-threads: 'false', file /builds/worker/checkouts/gecko/xpcom/threads/ThreadEventTarget.cpp:65

PCOM complains when events are dispatched during shutdown. Since we're
shutting down, there's no really any point in dispatching these messages
anyway, so we can safely ignore them.

Pushed by asferro@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d4f4dc5411d8
Don't try to dispatch XPCOM events during shutdown. r=aklotz
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 90 Branch
Has Regression Range: --- → yes
See Also: → 1774310
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: