1464938 - Crash in AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: *

Status: ASSIGNED

(Toolkit :: Add-ons Manager, defect, P2)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is
report bp-839f0d14-e01c-441a-9565-9b40e0180528.
=============================================================

Seen while looking at Mac crash stats - these crashes started using 20180527220122. Several unique users appear to have hit this crash.

Possible regression range based on Build ID: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3eccd139667d491ca259e8ff96f740ecfa36a781&tochange=db78be8dbc1c81844eb7d35c1a3073078eb5d923

ni on :kmag since he landed a bunch of stuff in that timeframe

Top 10 frames of crashing thread:

0 libmozglue.dylib mozalloc_abort memory/mozalloc/mozalloc_abort.cpp:34
1 XUL NS_DebugBreak xpcom/base/nsDebugImpl.cpp:457
2 XUL nsDebugImpl::Abort xpcom/base/nsDebugImpl.cpp:146
3 XUL NS_InvokeByIndex 
4 XUL XPCWrappedNative::CallMethod js/xpconnect/src/XPCWrappedNative.cpp:1707
5 XUL XPC_WN_CallMethod js/xpconnect/src/XPCWrappedNativeJSOps.cpp:899
6 XUL js::InternalCallOrConstruct js/src/vm/JSContext-inl.h:279
7 XUL Interpret js/src/vm/Interpreter.cpp:526
8 XUL js::RunScript js/src/vm/Interpreter.cpp:421
9 XUL js::InternalCallOrConstruct js/src/vm/Interpreter.cpp:493

=============================================================

Comment 1

[Kris Maglione [:kmag]]

This is probably just because I removed the timeout that limited the shutdown blocker to only wait for a few seconds. My best guess is that those users have frozen content processes that stop responding but never get killed.

Comment 2

[Raul Nicolae Malea]

Crash again. Crash report: https://crash-stats.mozilla.com/report/index/a71e2a7b-5db3-4494-8ae4-8e1380180610

Comment 3

[[:philipp]]

i'm a bit confused over the state of this bug - it's marked as dupe of 1464743 which received a fix a month ago, but the crash volume hasn't gone down a bit.

this is one of the top regressions showing up in crash-stats data for the early 62.0b cycle. i'm reopening this report, so that we don't lose track of this issue.

Comment 4

[Kris Maglione [:kmag]]

Yeah, sorry. Bug 1464743 only fixed part of the problem. We now timeout waiting for responses from hung content process at extension shutdown, but not at extension startup. And since startup blocks shutdown...

Comment 5

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 62.

Comment 6

[[:philipp]]

counting all signatures, the issue is accounting for 10% of browser crashes on 62.0b6

Comment 7

[Mike Taylor [:miketaylr]]

Heya Kris, any updates here?

Comment 8

[Intermittent Failures Robot]

1 failures in 623 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 1

Platform breakdown:
* linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1464938&startday=2018-07-09&endday=2018-07-15&tree=trunk

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

David, this is our top crash and a new regression in beta 62, can you help find someone to investigate?

Comment 10

[Kris Maglione [:kmag]]

Attachment: Bug 1464938: Stop needlessly waiting for responses to startup messages.

Review commit: https://reviewboard.mozilla.org/r/259454/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/259454/

Comment 11

[Kris Maglione [:kmag]]

Note: This patch does not, of course, fix the underlying problem of content processes freezing. We still get shutdown timeouts, but they'll shift to:

WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"SessionStore: flushing all windows","state":{"total":1,"current":0},"filename":"resource:///modules/sessionstore/SessionStore.jsm","lineNumber":1619,"stack":["resource:///modules/sessionstore/SessionStore.jsm:ssi_onQuitApplicationGranted:1619","resource:///modules/sessionstore/SessionStore.jsm:ssi_observe:780","chrome://global/content/globalOverlay.js:goQuitApplication:56","chrome://browser/content/browser.xul:oncommand:1"]}] Barrier: quit-application-granted

Which is good, because it means someone else will get nagged about them rather than me.

But it also means that extensions that are updated while a content process is frozen will actually be able to start, which is good.

Comment 12

[Kris Maglione [:kmag]]

Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/259454/diff/1-2/

Comment 13

[Andrew Swan [:aswan]]

Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

https://reviewboard.mozilla.org/r/259454/#review266552

Comment 14

[Kris Maglione [:kmag]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c034e7ddbacdbe80ec8ce5b844e4299fcac9287c
Bug 1464938: Stop needlessly waiting for responses to startup messages. r=aswan

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/c034e7ddbacd

Comment 16

[David Durst [:ddurst]]

Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: Continued crashes on beta
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: not yet
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none (though subsequent shutdown timeouts will be directed to other bugs)
[Is the change risky?]: no
[Why is the change risky/not risky?]: we will no longer wait on startup messages from frozen content processes
[String changes made/needed]: none

Comment 17

[Pascal Chevrel:pascalc]

Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

Fix for a high volume crash, approval for beta 13.

Comment 18

[Narcis Beleuzu [:NarcisB]]

kmag, the patch fails to apply from beta because the underlying code got changed in bug 1470783 which didn't get uplifted. Please provide a patch for beta.

Comment 19

[[:philipp]]

there hasn't been a clear reduction of those crashes on nightly after the patch landed a week ago - not sure if we should reopen this bug or file a new one to cover the remaining work.

these signatures are the biggest stability regression showing up in crash-stats for firefox 62 at this point, accounting for 12% of browser crashes in b14.
there are still 3 weeks in the beta cycle before 62 goes to release candidate builds.

Comment 20

[Liz Henry (:lizzard) (relman/hg->git project)]

Emailing kmag to ask about uplift. Crash volume is so low in nightly that we might not see an effect. It might be still worth uplift to beta.

Comment 21

[Andreea Pavel [:apavel]]

Backed out for failing browser/components/extensions/test/browser/browser_ext_browserAction_popup_preload.js

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=52eadd214d8ba5315684e1b73873c7748a5c990b

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=192949362&repo=mozilla-beta&lineNumber=2161

Backout: https://hg.mozilla.org/releases/mozilla-beta/rev/c2c8c742203a6adbe29a4c3b4a5eb036417c7671

Comment 22

[Kris Maglione [:kmag]]

Added the fix for the races in that test from bug 1357487 and landed again.

https://hg.mozilla.org/releases/mozilla-beta/rev/ee7538f8ddf8

Comment 23

[Kris Maglione [:kmag]]

It looks like there are other causes of this that weren't fixed by my patches, so this should stay open for now.

Comment 24

[[:philipp]]

early indications from crash data on 62.0b17 suggests that these crashes are still present but considerably lower in volume (4.3% of browser crashes vs the 10-12% figure we had before).

Comment 25

[Mike Taylor [:miketaylr]]

Jim, do you know anyone who can take a look?

Comment 27

[Marcia Knous [:marcia]]

So although some of the signatures in this bug are relatively small, AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: [e-mail] has 4092 crashes. Although that is still small relative to the release population, should we look into why Firefox Monitor is crashing?

Comment 28

[David Durst [:ddurst]]

Absolutely, yes.

Comment 29

[(inactive) Nihanth Subramanya [:nhnt11]]

Attachment: fxmonitor: Don't bother cleaning up observers if we're shutting down.

The fxmonitor crash is happening because extension shutdown is not completing quickly enough, which prompts AsyncShutdown to crash.

The only thing I can think of that might cause this to happen is that there's a JavaScript error while the extension cleans up window listeners, causing the promise to not resolve, leading to the timeout being reached and the crash.

So far I have been unable to reproduce the crash, so this is proving a bit hard to debug "properly". I'll keep trying, but in the meantime, I strongly suspect that this will be fixed by ensuring [1] does not run if we are in async shutdown. This is the only code that is called during extension shutdown that might result in an error if (for example) Services.wm is not available, or if the windows are in a teardown state and we should not be messing with them.

In any case, that code does not need to run at async shutdown, it only exists to handle when the addon is being disabled. What I'm wondering though, is if it's okay to make a proactive change based on this prediction and push that already to see if it mitigates the crash spike (in lieu of definitive evidence).

[1] https://github.com/mozilla/blurts-addon/blob/6eaac3f80a85f5d17780dd8b5d77e99b95405760/src/privileged/subscripts/EveryWindow.jsm#L34

Comment 30

[Johann Hofmann [:johannh]]

Comment on attachment 9010687 [details] [diff] [review]
fxmonitor: Don't bother cleaning up observers if we're shutting down.

Review of attachment 9010687 [details] [diff] [review]:
-----------------------------------------------------------------

As mentioned on IRC, I can give a quick rs=me as I don't think this will be harmful, but I can't verify that it fixes the issue :)

Comment 31

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Nihanth, as Marcia mentioned in the email, we should push the new SAO update with an updated minor revision, so we are able to distinguish it from the crash reports?

Comment 32

[(inactive) Nihanth Subramanya [:nhnt11]]

(In reply to Ritu Kothari (:ritu) from comment #31)
> Hi Nihanth, as Marcia mentioned in the email, we should push the new SAO
> update with an updated minor revision, so we are able to distinguish it from
> the crash reports?

The build that I uploaded in bug 1486874 comment 22 does indeed bump the version number to 2.3.

Comment 33

[Kris Maglione [:kmag]]

Attachment: Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Comment 34

[Andrew Swan [:aswan]]

Comment on attachment 9010724 [details]
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Andrew Swan [:aswan] has approved the revision.

Comment 35

[Kris Maglione [:kmag]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e30f26e9a590588e3f5f2afa54900ee8f3d55190
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Comment 36

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/e30f26e9a590

Comment 37

[Pascal Chevrel:pascalc]

Kris, could you request beta uplift for your last patch? Thanks

Comment 38

[Marcia Knous [:marcia]]

Adding a new signature seen for fxmonitor.

Comment 39

[Marcia Knous [:marcia]]

Also a correlation that may be helpful, seen in fxmonitor signature: (100.0% in signature vs 06.17% overall) abort_message = ###!!! ABORT: file resource://gre/modules/addons/XPIProvider.jsm, line 2192

Comment 40

[Kris Maglione [:kmag]]

Comment on attachment 9010724 [details]
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Approval Request Comment
[Feature/Bug causing the regression]: N/A
[User impact if declined]: This patch will help us diagnose common shutdown hangs which we so far have not been able to get any sort of handle on. The crash rates aren't high enough in nightly to get much useful information.
[Is this code covered by automated tests?]: Yes, but the added diagnostics are not.
[Has the fix been verified in Nightly?]: Yes, we are getting the expected data in crashstats, just at a very low rate.
[Needs manual test from QE? If yes, steps to reproduce]: No.
[List of other uplifts needed for the feature/fix]: None.
[Is the change risky?]: No.
[Why is the change risky/not risky?]: It simply adds some rather trivial diagnostic labels to the extension startup/shutdown flows.
[String changes made/needed]: None.

Comment 41

[Pascal Chevrel:pascalc]

Comment on attachment 9010724 [details]
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Uplift approved for 63 beta 10, thanks.

Comment 42

[(PTO June 19-July 7) Ryan VanderMeulen]

https://hg.mozilla.org/releases/mozilla-beta/rev/ba673d83c7c4

Comment 44

[Rob Wu [:robwu]]

The extra information reported via comment 36 and comment 42 is very helpful; I'm going to try and fix the remaining crashes.

In the past month:

68303 with unknown crash state ( "(none)" ) (5893 in the past week).
2998 with known crash state (1536 in past week )

 1. 1886 at asyncEmitManifestEntry(\"chrome_settings_overrides\") @ bug 1493770
 2.  402 at asyncEmitManifestEntry(\"background\") @ bug 1501375
 3.  743 at "Startup: Run manifest\""
 4.  31 at "Startup: Loading manifest\""
 5.  21 at "Shutdown: Flushed jar cache"
 6.  16 at "Shutdown: Emit shutdown"
 7.  8 at asyncEmitManifestEntry(\"commands\")
 8.  6 at "Shutdown: Storage"
 9.  6 at asyncEmitManifestEntry(\"chrome_url_overrides\")
10.  1 at asyncEmitManifestEntry(\"browser_action\")
11.  1 at "Shutdown: Flush jar cache"
(those numbers add up to 3121, which is more than 2998 because some crashes contain multiple add-on states).

Links:
 1. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry(%5C%5C%5C%22chrome_settings_overrides%5C%5C%5C%22)&date=%3E%3D2018-10-01T00%3A00%3A00.000Z&date=%3C2018-11-01T00%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 2. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22background%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 3. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Run%20manifest%5C%22&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 4. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Loading%20manifest%5C%22&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 5. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flushed%20jar%20cache&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 6. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Emit%20shutdown&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 7. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22commands%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 8. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Storage&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 9. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22chrome_url_overrides%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
10. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22browser_action%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
11. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flush%20jar%20cache&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports

This is the query that excludes all of the above (should not have any results, because I tried to find all potential causes):
https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=%21%40.%2A%28%5C%28none%5C%29%7C%5C%5C%5C%22chrome_settings_overrides%5C%5C%5C%22%7C%5C%5C%5C%22background%5C%5C%5C%22%7C%5C%5C%5C%22commands%5C%5C%5C%22%7C%5C%5C%5C%22browser_action%5C%5C%5C%22%7C%5C%5C%5C%22chrome_url_overrides%5C%5C%5C%22%7CStartup%3A%20Run%20manifest%5C%22%7CStartup%3A%20Loading%20manifest%5C%22%7CShutdown%3A%20Emit%20shutdown%7CShutdown%3A%20Storage%7CShutdown%3A%20Flushed%20jar%20cache%7CShutdown%3A%20Flush%20jar%20cache%29.%2A&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports

Comment 45

[Mike Kaply [:mkaply]]

Assuming this:

https://crash-stats.mozilla.com/report/index/db39437d-2d42-4e58-9d33-a13e90181116

is the same crash (which it looks like), I have a recreation scenario on Windows.

Create a new profile and make sure it doesn't show the "multiple tabs open" message on shutdown.

1. Use the web page method to install an add-on which includes a search engine (eg. https://ext.ask.com/index.jhtml)
2. When the second prompt is showing, which is for the default search, close Firefox using the X button in the window title bar

If anything interferes with the closing of Firefox, it won't happen (dialog popup).

What you'll see is that multiple processes are left open in the task manager and after a minute or so, Firefox crashes.

Comment 46

[Kris Maglione [:kmag]]

(In reply to Mike Kaply [:mkaply] from comment #45)
> Assuming this:
> 
> https://crash-stats.mozilla.com/report/index/db39437d-2d42-4e58-9d33-
> a13e90181116
> 
> is the same crash (which it looks like), I have a recreation scenario on
> Windows.
> 
> Create a new profile and make sure it doesn't show the "multiple tabs open"
> message on shutdown.
> 
> 1. Use the web page method to install an add-on which includes a search
> engine (eg. https://ext.ask.com/index.jhtml)

That is bug 1493770, which is assigned to you.

Comment 49

[Neha Kochar [:neha]]

Rob, are the the remaining crashes on target to get a fix for 67?

Comment 50

[Rob Wu [:robwu]]

No, the others are not in 67.

The top cause (asyncEmitManifestEntry(\"chrome_settings_overrides\")) was fixed by bug 1525729, just before the weekend, in time for Firefox 67.

The second top cause (asyncEmitManifestEntry(\"background\")) was supposedly addressed by bug 1501375, but there are still crash reports with the signature.

The others haven't been addressed yet. Each of the others (from comment 44) should be tracked separately, but I'm keeping them in this one bug report since the crash signatures are indistinguishable (the relevant information is in the async shutdown timeout field, which is not recognized by BMO).

Comment 51

[Rob Wu [:robwu]]

I suspect that many of these crashes occur when the lengthy extension startup continues after shutdown has already been requested, as 0.4% of the startups reported by telemetry take over 50 seconds (0.4% on Beta 67, 0.5% on Nightly 68).
Adding a few checks and early returns from the extension starup could reduce the number of timeouts, but there will always be some timeouts under unfortunate circumstances on constrained systems (e.g. slow file access).

The page_action handler is a clear example that couldn't have been the cause of a timeout. It awaits StartupCache.get, but at that point the startup cache has already been read (to read extension manifests), so the call is expected to resolve within a few microtasks. Yet there were 10 AsyncShutdown timeout crashes associated with page_action in the past three months, 9 of these are from one of the built-in extensions (screenshots, webcompat).

I filed two new dependencies (bug 1543354 and bug 1542181).
Bug 1543354 is actionable, whereas bug 1542181 is just a report of the issue with no clear path forwards, as it is likely dependent on file IO.
The crash rates for the remaining signatures could be lowered by adding early returns to the Extension's startup if shutdown has started.

---

For comparison, update to comment 44:

1. 1272 at asyncEmitManifestEntry(\"chrome_settings_overrides\") (1 after bug 1525729 landed)
2. 499 at asyncEmitManifestEntry(\"background\") - bug 1543354
3. 933 at at "Startup: Run manifest\""
4. 27 at "Startup: Loading manifest\""
5. 30 at "Shutdown: Flushed jar cache"
6. 21 at "Shutdown: Emit shutdown"
7. 15 at asyncEmitManifestEntry(\"commands\")
8. 351 at "Shutdown: Storage" (significant regression - see bug 1542181)
9. 2 at asyncEmitManifestEntry(\"chrome_url_overrides\")
10. 0 at asyncEmitManifestEntry(\"browser_action\") (7 in past three months)
11. 0 at "Shutdown: Flush jar cache" (2 in past three months)
12. 0 at asyncEmitManifestEntry(\"page_action\") (10 in past three months) (NEW)

1. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry(%5C%5C%5C%22chrome_settings_overrides%5C%5C%5C%22)&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
2. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22background%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
3. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Run%20manifest%5C%22&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
4. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Loading%20manifest%5C%22&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
5. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flushed%20jar%20cache&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
6. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Emit%20shutdown&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
7. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22commands%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
8. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Storage&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
9. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22chrome_url_overrides%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
10. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22browser_action%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
11. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flush%20jar%20cache&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
12. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22page_action%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&page=1&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports

Comment 54

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

I get this crash today [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: [e-mail] ] - https://crash-stats.mozilla.org/report/index/87625ef7-5b31-4a3b-9589-e95240200427

Comment 55

[Robert Hartmann]

Reproducible with
Product Thunderbird
Release Channel nightly
Version 83.0a1
Build ID 20200922082006 (2020-09-22)

Just start Thunderbird ... doing nothing ... and close main TB window ... wait a while and crash:

Thunderbird 83.0a1 Crash Report [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: [e-mail] ]

32bit crash : bp-f5728164-216f-4f8e-a14d-5aae20200922
64bit crash : bp-3078bd25-dfd3-472d-9728-678ee0200922

see bug 1559448 comment 9

Comment 56

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:robwu, maybe it's time to close this bug?

Comment 58

[Rob Wu [:robwu]]

The vast majority of the crashes related to this are annotated with asyncEmitManifestEntry(\"background\"), which is tracked in bug 1543354.

In the past month, Crash in AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: has:

• 622 crashes without asyncEmitManifestEntry(\"background\")- https://crash-stats.mozilla.org/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=%21~asyncEmitManifestEntry%28%5C%5C%5C%22background%5C%5C%5C%22%29&product=Firefox&date=%3E%3D2021-03-29T17%3A13%3A00.000Z&date=%3C2021-04-29T17%3A13%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
• 4693 crashes with asyncEmitManifestEntry(\"background\") - https://crash-stats.mozilla.org/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22background%5C%5C%5C%22%29&product=Firefox&date=%3E%3D2021-03-29T17%3A13%3A00.000Z&date=%3C2021-04-29T17%3A13%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports

Comment 62

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on release

For more information, please visit auto_nag documentation.

Comment 63

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit auto_nag documentation.

Comment 65

[BugBot [:suhaib / :marco/ :calixte]]

Since the crash volume is low (less than 15 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.

For more information, please visit BugBot documentation.