1674388 - [macOS] Crash in [@ IPCError-browser | ShutDownKill | __psynch_cvwait | mozilla::TaskController::GetRunnableForMTTask | mozilla::ipc::MessagePump::Run]

Status: RESOLVED DUPLICATE of bug 1279293

(Core :: XPCOM, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Only affects Nightly, and >=80.0a1.

Maybe Fission related. (DOMFissionEnabled=1)

Crash report: https://crash-stats.mozilla.org/report/index/ef627e47-6a28-47ba-b19e-e10180201030

Reason: EXC_BREAKPOINT / EXC_I386_BPT

Top 10 frames of crashing thread:

0 libsystem_kernel.dylib __psynch_cvwait 
1 libmozglue.dylib mozilla::detail::ConditionVariableImpl::wait mozglue/misc/ConditionVariable_posix.cpp:108
2 XUL mozilla::TaskController::GetRunnableForMTTask xpcom/threads/TaskController.cpp:283
3 XUL nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1143
4 XUL mozilla::ipc::MessagePump::Run ipc/glue/MessagePump.cpp:109
5 XUL MessageLoop::Run ipc/chromium/src/base/message_loop.cc:309
6 XUL nsBaseAppShell::Run widget/nsBaseAppShell.cpp:137
7 XUL nsAppShell::Run widget/cocoa/nsAppShell.mm:694
8 XUL XRE_RunAppShell toolkit/xre/nsEmbedFunctions.cpp:913
9 XUL MessageLoop::Run ipc/chromium/src/base/message_loop.cc:309

Comment 1

[Chris Peterson [:cpeterson]]

Appears to be a regression in 80.0a1. The first crashing build ID for this crash signature is 20200726095051:

https://crash-stats.mozilla.org/search/?signature=~IPCError-browser%20%7C%20ShutDownKill%20%7C%20__psynch_cvwait%20%7C%20mozilla%3A%3ATaskController%3A%3AGetRunnableForMTTask%20%7C%20mozilla%3A%3Aipc%3A%3AMessagePump%3A%3ARun&version=80.0a1&product=Firefox&date=%3E%3D2020-01-25T21%3A46%3A00.000Z&date=%3C2020-11-25T21%3A46%3A00.000Z&_facets=signature&_facets=build_id&_facets=version&_facets=dom_fission_enabled&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-build_id

I'm not sure why the Crash Data graph above shows no crashes before 2020-08-21.

This crash is Nightly-only but it's not a Fission bug: only 11% of the crash reports have Fission enabled:

https://crash-stats.mozilla.org/search/?signature=~IPCError-browser%20%7C%20ShutDownKill%20%7C%20__psynch_cvwait%20%7C%20mozilla%3A%3ATaskController%3A%3AGetRunnableForMTTask%20%7C%20mozilla%3A%3Aipc%3A%3AMessagePump%3A%3ARun&product=Firefox&date=%3E%3D2020-01-25T21%3A46%3A00.000Z&date=%3C2020-11-25T21%3A46%3A00.000Z&_facets=signature&_facets=build_id&_facets=version&_facets=dom_fission_enabled&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-dom_fission_enabled

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Change the status for beta to have the same as nightly and release.
For more information, please visit auto_nag documentation.

Comment 3

[Pascal Chevrel:pascalc]

The volume of this crash on nightly has tripled since last month, could that be the related to the wider rollout of FIssion?

Comment 4

[Andrew McCreight [:mccr8]]

Around 85% of the most common signature, IPCError-browser | ShutDownKill | __psynch_cvwait | mozilla::TaskController::GetRunnableForMTTask | mozilla::ipc::MessagePump::Run, have Fission enabled.

Despite these being mostly Fission crashes, I don't see a single crash where there is a remote type annotation on the crash report, which I think implies that ContentChild::RecvRemoteType has never run, which in turn I think implies that these are preallocated processes.

Comment 5

[Andrew McCreight [:mccr8]]

Maybe there's some kind of race where the preallocated process manager is in the middle of starting up a new process? I'm not sure why the preallocated process manager doesn't call CloseProcesses() when it observes that shutdown has begun. It looks like the behavior was intentionally changed to this in bug 1363601. It looks like ContentParent::ReleaseCachedProcesses() might clear those out, but I don't understand how that is invoked.

Comment 6

[Andrew McCreight [:mccr8]]

For what it is worth, the last 5 crashes on my OSX machine looked like this.

Comment 7

[Chris Peterson [:cpeterson]]

(In reply to Pascal Chevrel:pascalc from comment #3)

The volume of this crash on nightly has tripled since last month, could that be the related to the wider rollout of FIssion?

(In reply to Andrew McCreight [:mccr8] from comment #4)

Despite these being mostly Fission crashes, I don't see a single crash where there is a remote type annotation on the crash report, which I think implies that ContentChild::RecvRemoteType has never run, which in turn I think implies that these are preallocated processes.

CC'ing Jesup because these ShutDownKills look related to Fission's pre-allocated processes.

Tracking this bug for Fission M7a because we should understand this issue before we enable Fission for more users.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

I'll investigate. Fission has 3x the number of preallocated processes -- and more to the point perhaps is far more likely to be in the process of preallocating a process when we shut down.

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The signature changed from IPCError-browser | ShutDownKill | __psynch_cvwait | mozilla::TaskController::GetRunnableForMTTask | mozilla::ipc::MessagePump::Run to [@ IPCError-browser | ShutDownKill | __psynch_cvwait | mozilla::ipc::MessagePump::Run]. The signature for the latter got also added to bug 1279293.

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

This subset of the ShutDownKill errors is a mac-only, and the particular signature is likely related to process startup (given the lack of a remote type - not even Web or Prealloc). Since we're allocating processes more often, this overall makes sense.

This causes no significant negative impact to users, so pushing this out.

Comment 11

[Neha Kochar [:neha]]

Dup'ing as discussed with Randell in the Fission meeting.