1387369 - Crash in AsyncShutdownTimeout | profile-before-change | Crash Reporter: blocking on minidumpgeneration.

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-b7ab3852-ca97-41da-a4e7-5d0ca0170804.
=============================================================
AsyncShutdownTimeout 	
{"phase":"profile-before-change","conditions":[{"name":"Crash Reporter: blocking on minidumpgeneration.","state":"(none)","filename":"z:/build/build/src/ipc/glue/CrashReporterHost.cpp","lineNumber":192,"stack":"Minidump generation"}]}

this async shutdown timeout crash was introduced with bug 1360308. 
it is in the top 5 browser crashes of 57.0a1 in its first few days.

Comment 1

[Cervantes Yu [:cyu] [:cervantes]]

Looks like minidump generation can be so slow as to cause shutdown timeout. This is probably what happened:

* A child process hangs.
* The parent process detects that and generates a minidump for that.
* The user closes the browser, but we block shutdown during the generation of minidumps.
* Shutdown timeout crash.

I am not sure whether we can cleanly unblock the above scenario. If we don't block shutdown on minidump generation, we can still block in shutting down the thread for async dump generation: https://dxr.mozilla.org/mozilla-central/rev/52285ea5e54c73d3ed824544cef2ee3f195f05e6/toolkit/crashreporter/nsExceptionHandler.cpp#3568

Or we just don't shut down the thread when we have pending operation on it during process shutdown and let the thread leak, but it doesn't sound good that we let the thread continue writing the minidump while we exit the main thread.

Comment 2

[Jim Mathies [:jimm]]

shutdown timeout auto generated crash.

Comment 3

[Jonathan Howard]

Pre async crashes were in bug 1182927.
Increased volume likely down to a shorter timeout.

Comment 4

[Trevor Skywalker]

Top Crasher:
#10 in Release.
#40 in Beta.
#83 in Nightly.

Top Crashers for Firefox 60.0
Top 50 Crashing Signatures. 7 days ago	
	
10 	1.03% 	0.97% 	AsyncShutdownTimeout | profile-before-change | Crash Reporter: blocking on minidumpgeneration.	432 	123 	1 	2 	247 	0 	2017-06-24

Top Crashers for Firefox 61.0b
Top 50 Crashing Signatures. 7 days ago

40 	0.27% 	-0.17% 	AsyncShutdownTimeout | profile-before-change | Crash Reporter: blocking on minidumpgeneration.	78 	75 	0 	0 	75 	0 	2017-06-24

Top Crashers for Firefox 62.0a1
Top 300 Crashing Signatures. 7 days ago

83 	0.15% 	0.04% 	AsyncShutdownTimeout | profile-before-change | Crash Reporter: blocking on minidumpgeneration.	6 	4 	0 	0 	3 	0 	2017-06-24

Comment 5

[Mike Taylor [:miketaylr]]

Lots of comments about using firejail on linux.

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Mike Taylor [:miketaylr] (62 Regression Engineering Owner) from comment #5)
> Lots of comments about using firejail on linux.

Interesting, I wonder if that's related to bug 1461848.

Comment 7

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Bug 1461848 seems to cause minidump generation to fail immediately, but firejail is different from the Snap sandbox.  I've just tried using firejail (from the package in Debian unstable) with a mozilla-central build, and it seems to work, but of course there are a lot of other variables here.

The comments that indicate not being able to start any content processes are worrying (and they seem to all be on Linux?), but there isn't much information in those metadata-only crash reports.

Comment 8

[Pascal Chevrel:pascalc (PTO until April 26)]

No crashes since 63 shipped, the remaining crashes we still have are people on ESR. Adjusting flags accordingly.

Comment 9

[Trevor Skywalker]

Was this bug fixed by changes implemented in Bug 1463048?