Open Bug 1464938 Opened 2 years ago Updated 2 months ago

Crash in AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: *

Categories

(Toolkit :: Add-ons Manager, defect, P2, critical)

62 Branch
defect

Tracking

()

ASSIGNED
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- affected
firefox61 --- unaffected
firefox62 + wontfix
firefox63 --- wontfix
firefox64 --- wontfix
firefox65 --- wontfix
firefox66 --- wontfix
firefox67 --- wontfix
firefox68 --- fix-optional

People

(Reporter: marcia, Assigned: robwu)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Keywords: crash, leave-open, regression)

Crash Data

Attachments

(2 files, 1 obsolete file)

This bug was filed from the Socorro interface and is
report bp-839f0d14-e01c-441a-9565-9b40e0180528.
=============================================================

Seen while looking at Mac crash stats - these crashes started using 20180527220122. Several unique users appear to have hit this crash.

Possible regression range based on Build ID: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3eccd139667d491ca259e8ff96f740ecfa36a781&tochange=db78be8dbc1c81844eb7d35c1a3073078eb5d923

ni on :kmag since he landed a bunch of stuff in that timeframe

Top 10 frames of crashing thread:

0 libmozglue.dylib mozalloc_abort memory/mozalloc/mozalloc_abort.cpp:34
1 XUL NS_DebugBreak xpcom/base/nsDebugImpl.cpp:457
2 XUL nsDebugImpl::Abort xpcom/base/nsDebugImpl.cpp:146
3 XUL NS_InvokeByIndex 
4 XUL XPCWrappedNative::CallMethod js/xpconnect/src/XPCWrappedNative.cpp:1707
5 XUL XPC_WN_CallMethod js/xpconnect/src/XPCWrappedNativeJSOps.cpp:899
6 XUL js::InternalCallOrConstruct js/src/vm/JSContext-inl.h:279
7 XUL Interpret js/src/vm/Interpreter.cpp:526
8 XUL js::RunScript js/src/vm/Interpreter.cpp:421
9 XUL js::InternalCallOrConstruct js/src/vm/Interpreter.cpp:493

=============================================================
Flags: needinfo?(kmaglione+bmo)
This is probably just because I removed the timeout that limited the shutdown blocker to only wait for a few seconds. My best guess is that those users have frozen content processes that stop responding but never get killed.
Status: NEW → RESOLVED
Closed: 2 years ago
Flags: needinfo?(kmaglione+bmo)
Resolution: --- → DUPLICATE
Duplicate of bug: 1464743
i'm a bit confused over the state of this bug - it's marked as dupe of 1464743 which received a fix a month ago, but the crash volume hasn't gone down a bit.

this is one of the top regressions showing up in crash-stats data for the early 62.0b cycle. i'm reopening this report, so that we don't lose track of this issue.
Status: RESOLVED → REOPENED
Has Regression Range: --- → yes
OS: Mac OS X → All
Hardware: Unspecified → All
Resolution: DUPLICATE → ---
Version: unspecified → 62 Branch
Yeah, sorry. Bug 1464743 only fixed part of the problem. We now timeout waiting for responses from hung content process at extension shutdown, but not at extension startup. And since startup blocks shutdown...
Assignee: nobody → kmaglione+bmo
Tracking for 62.
counting all signatures, the issue is accounting for 10% of browser crashes on 62.0b6
Crash Signature: [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: @testpilot-addon] → [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: @testpilot-addon] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}] [@ AsyncShutdownTimeout | profile-change-teardown |…
Summary: Crash in AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: @testpilot-addon → Crash in AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: *
Crash Signature: {4ED1F68A-5463-4931-9384-8FFF5ED91D92}] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: wrc@avast.com] → {4ED1F68A-5463-4931-9384-8FFF5ED91D92}] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: wrc@avast.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: coba@mozilla.com.cn,Extension shutdown: easyscreensho…
Heya Kris, any updates here?
Flags: needinfo?(kmaglione+bmo)
Crash Signature: {4ED1F68A-5463-4931-9384-8FFF5ED91D92}] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: wrc@avast.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: coba@mozilla.com.cn,Extension shutdown: easyscreensho… → {4ED1F68A-5463-4931-9384-8FFF5ED91D92}] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: wrc@avast.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: sp@avast.com,Extension shutdown: wrc@avast.com [@ As…
Crash Signature: ,Extension shutdown: wrc@avast.com [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: coba@mozilla.com.cn,Extension shutdown: easyscreenshot@mozillaonline.com,Extension shutdown: tabtweak@mozillaonline.com → ,Extension shutdown: wrc@avast.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: coba@mozilla.com.cn,Extension shutdown: easyscreenshot@mozillaonline.com,Extension shutdown: tabtweak@mozillaonline.com
Crash Signature: AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: _ceMembers_@free.easypdfcombine.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: {b9acf540-acba-11e1-8ccb-001fd0e08bd4}] → AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: _ceMembers_@free.easypdfcombine.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: {b9acf540-acba-11e1-8ccb-001fd0e08bd4}] [@ AsyncShutdownTimeout | profile-…
David, this is our top crash and a new regression in beta 62, can you help find someone to investigate?
Flags: needinfo?(ddurst)
Note: This patch does not, of course, fix the underlying problem of content processes freezing. We still get shutdown timeouts, but they'll shift to:

WARNING: At least one completion condition is taking too long to complete. Conditions: [{"name":"SessionStore: flushing all windows","state":{"total":1,"current":0},"filename":"resource:///modules/sessionstore/SessionStore.jsm","lineNumber":1619,"stack":["resource:///modules/sessionstore/SessionStore.jsm:ssi_onQuitApplicationGranted:1619","resource:///modules/sessionstore/SessionStore.jsm:ssi_observe:780","chrome://global/content/globalOverlay.js:goQuitApplication:56","chrome://browser/content/browser.xul:oncommand:1"]}] Barrier: quit-application-granted

Which is good, because it means someone else will get nagged about them rather than me.

But it also means that extensions that are updated while a content process is frozen will actually be able to start, which is good.
Flags: needinfo?(kmaglione+bmo)
Flags: needinfo?(ddurst)
Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

https://reviewboard.mozilla.org/r/259454/#review266552
Attachment #8994957 - Flags: review?(aswan) → review+
https://hg.mozilla.org/mozilla-central/rev/c034e7ddbacd
Status: REOPENED → RESOLVED
Closed: 2 years agoLast year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

Approval Request Comment
[Feature/Bug causing the regression]: n/a
[User impact if declined]: Continued crashes on beta
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: not yet
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none (though subsequent shutdown timeouts will be directed to other bugs)
[Is the change risky?]: no
[Why is the change risky/not risky?]: we will no longer wait on startup messages from frozen content processes
[String changes made/needed]: none
Attachment #8994957 - Flags: approval-mozilla-beta?
Comment on attachment 8994957 [details]
Bug 1464938: Stop needlessly waiting for responses to startup messages.

Fix for a high volume crash, approval for beta 13.
Attachment #8994957 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
kmag, the patch fails to apply from beta because the underlying code got changed in bug 1470783 which didn't get uplifted. Please provide a patch for beta.
Flags: needinfo?(kmaglione+bmo)
there hasn't been a clear reduction of those crashes on nightly after the patch landed a week ago - not sure if we should reopen this bug or file a new one to cover the remaining work.

these signatures are the biggest stability regression showing up in crash-stats for firefox 62 at this point, accounting for 12% of browser crashes in b14.
there are still 3 weeks in the beta cycle before 62 goes to release candidate builds.
Flags: needinfo?(ddurst)
Emailing kmag to ask about uplift. Crash volume is so low in nightly that we might not see an effect. It might be still worth uplift to beta.
Added the fix for the races in that test from bug 1357487 and landed again.

https://hg.mozilla.org/releases/mozilla-beta/rev/ee7538f8ddf8
Status: RESOLVED → REOPENED
Flags: needinfo?(kmaglione+bmo)
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: Last yearLast year
Resolution: --- → FIXED
It looks like there are other causes of this that weren't fixed by my patches, so this should stay open for now.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Crash Signature: profile-change-teardown | Extension shutdown: federated-learning@shield.mozilla.org] → profile-change-teardown | Extension shutdown: federated-learning@shield.mozilla.org] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: YoutubeDownloader@PeterOlayev.com]
early indications from crash data on 62.0b17 suggests that these crashes are still present but considerably lower in volume (4.3% of browser crashes vs the 10-12% figure we had before).
Depends on: 1483641
Crash Signature: profile-change-teardown | Extension shutdown: federated-learning@shield.mozilla.org] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: YoutubeDownloader@PeterOlayev.com] → profile-change-teardown | Extension shutdown: federated-learning@shield.mozilla.org] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: YoutubeDownloader@PeterOlayev.com] [@ AsyncShutdownTimeout | profile-change-teardown | Extensio…
Flags: needinfo?(ddurst)
See Also: → 1464743
Jim, do you know anyone who can take a look?
Flags: needinfo?(jmathies)
Crash Signature: Extension shutdown: search@mail.ru] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: 383882@modext.tech] → Extension shutdown: search@mail.ru] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: 383882@modext.tech] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: taarexpv3@shield.mozilla.org]
Duplicate of this bug: 1464743
Assignee: kmaglione+bmo → nobody
Crash Signature: Extension shutdown: search@mail.ru] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: 383882@modext.tech] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: taarexpv3@shield.mozilla.org] → Extension shutdown: search@mail.ru] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: 383882@modext.tech] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: taarexpv3@shield.mozilla.org] [@ AsyncShutdownTimeo…
So although some of the signatures in this bug are relatively small, AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org has 4092 crashes. Although that is still small relative to the release population, should we look into why Firefox Monitor is crashing?
Absolutely, yes.
The fxmonitor crash is happening because extension shutdown is not completing quickly enough, which prompts AsyncShutdown to crash.

The only thing I can think of that might cause this to happen is that there's a JavaScript error while the extension cleans up window listeners, causing the promise to not resolve, leading to the timeout being reached and the crash.

So far I have been unable to reproduce the crash, so this is proving a bit hard to debug "properly". I'll keep trying, but in the meantime, I strongly suspect that this will be fixed by ensuring [1] does not run if we are in async shutdown. This is the only code that is called during extension shutdown that might result in an error if (for example) Services.wm is not available, or if the windows are in a teardown state and we should not be messing with them.

In any case, that code does not need to run at async shutdown, it only exists to handle when the addon is being disabled. What I'm wondering though, is if it's okay to make a proactive change based on this prediction and push that already to see if it mitigates the crash spike (in lieu of definitive evidence).

[1] https://github.com/mozilla/blurts-addon/blob/6eaac3f80a85f5d17780dd8b5d77e99b95405760/src/privileged/subscripts/EveryWindow.jsm#L34
Assignee: nobody → nhnt11
Attachment #9010687 - Flags: review?(jhofmann)
Comment on attachment 9010687 [details] [diff] [review]
fxmonitor: Don't bother cleaning up observers if we're shutting down.

Review of attachment 9010687 [details] [diff] [review]:
-----------------------------------------------------------------

As mentioned on IRC, I can give a quick rs=me as I don't think this will be harmful, but I can't verify that it fixes the issue :)
Attachment #9010687 - Attachment is patch: true
Attachment #9010687 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #9010687 - Flags: review?(jhofmann) → review+
Hi Nihanth, as Marcia mentioned in the email, we should push the new SAO update with an updated minor revision, so we are able to distinguish it from the crash reports?
Flags: needinfo?(nhnt11)
Flags: needinfo?(mozillamarcia.knous)
(In reply to Ritu Kothari (:ritu) from comment #31)
> Hi Nihanth, as Marcia mentioned in the email, we should push the new SAO
> update with an updated minor revision, so we are able to distinguish it from
> the crash reports?

The build that I uploaded in bug 1486874 comment 22 does indeed bump the version number to 2.3.
Flags: needinfo?(nhnt11)
Flags: needinfo?(mozillamarcia.knous)
Flags: needinfo?(jmathies)
Keywords: leave-open
Comment on attachment 9010724 [details]
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Andrew Swan [:aswan] has approved the revision.
Attachment #9010724 - Flags: review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/e30f26e9a590588e3f5f2afa54900ee8f3d55190
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan
Kris, could you request beta uplift for your last patch? Thanks
Flags: needinfo?(kmaglione+bmo)
Depends on: 1493770
Adding a new signature seen for fxmonitor.
Crash Signature: AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org] → AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org,Extension shutdown: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} ]
Also a correlation that may be helpful, seen in fxmonitor signature: (100.0% in signature vs 06.17% overall) abort_message = ###!!! ABORT: file resource://gre/modules/addons/XPIProvider.jsm, line 2192
Comment on attachment 9010724 [details]
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Approval Request Comment
[Feature/Bug causing the regression]: N/A
[User impact if declined]: This patch will help us diagnose common shutdown hangs which we so far have not been able to get any sort of handle on. The crash rates aren't high enough in nightly to get much useful information.
[Is this code covered by automated tests?]: Yes, but the added diagnostics are not.
[Has the fix been verified in Nightly?]: Yes, we are getting the expected data in crashstats, just at a very low rate.
[Needs manual test from QE? If yes, steps to reproduce]: No.
[List of other uplifts needed for the feature/fix]: None.
[Is the change risky?]: No.
[Why is the change risky/not risky?]: It simply adds some rather trivial diagnostic labels to the extension startup/shutdown flows.
[String changes made/needed]: None.
Flags: needinfo?(kmaglione+bmo)
Attachment #9010724 - Flags: approval-mozilla-beta?
Comment on attachment 9010724 [details]
Bug 1464938: Add extension lifecycle state information to shutdown blocker for better diagnostics. r=aswan

Uplift approved for 63 beta 10, thanks.
Attachment #9010724 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Duplicate of this bug: 1420145
Depends on: 1501375
The extra information reported via comment 36 and comment 42 is very helpful; I'm going to try and fix the remaining crashes.

In the past month:

68303 with unknown crash state ( "(none)" ) (5893 in the past week).
2998 with known crash state (1536 in past week )

 1. 1886 at asyncEmitManifestEntry(\"chrome_settings_overrides\") @ bug 1493770
 2.  402 at asyncEmitManifestEntry(\"background\") @ bug 1501375
 3.  743 at "Startup: Run manifest\""
 4.  31 at "Startup: Loading manifest\""
 5.  21 at "Shutdown: Flushed jar cache"
 6.  16 at "Shutdown: Emit shutdown"
 7.  8 at asyncEmitManifestEntry(\"commands\")
 8.  6 at "Shutdown: Storage"
 9.  6 at asyncEmitManifestEntry(\"chrome_url_overrides\")
10.  1 at asyncEmitManifestEntry(\"browser_action\")
11.  1 at "Shutdown: Flush jar cache"
(those numbers add up to 3121, which is more than 2998 because some crashes contain multiple add-on states).

Links:
 1. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry(%5C%5C%5C%22chrome_settings_overrides%5C%5C%5C%22)&date=%3E%3D2018-10-01T00%3A00%3A00.000Z&date=%3C2018-11-01T00%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 2. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22background%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 3. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Run%20manifest%5C%22&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 4. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Loading%20manifest%5C%22&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 5. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flushed%20jar%20cache&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 6. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Emit%20shutdown&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 7. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22commands%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 8. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Storage&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
 9. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22chrome_url_overrides%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
10. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22browser_action%5C%5C%5C%22%29&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
11. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flush%20jar%20cache&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports

This is the query that excludes all of the above (should not have any results, because I tried to find all potential causes):
https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=%21%40.%2A%28%5C%28none%5C%29%7C%5C%5C%5C%22chrome_settings_overrides%5C%5C%5C%22%7C%5C%5C%5C%22background%5C%5C%5C%22%7C%5C%5C%5C%22commands%5C%5C%5C%22%7C%5C%5C%5C%22browser_action%5C%5C%5C%22%7C%5C%5C%5C%22chrome_url_overrides%5C%5C%5C%22%7CStartup%3A%20Run%20manifest%5C%22%7CStartup%3A%20Loading%20manifest%5C%22%7CShutdown%3A%20Emit%20shutdown%7CShutdown%3A%20Storage%7CShutdown%3A%20Flushed%20jar%20cache%7CShutdown%3A%20Flush%20jar%20cache%29.%2A&date=%3E%3D2018-10-01T02%3A00%3A00.000Z&date=%3C2018-11-01T01%3A00%3A00.000Z&page=1&_sort=-date&_facets=signature&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
Assignee: nhnt11 → rob
Status: REOPENED → ASSIGNED
Priority: -- → P2
Assuming this:

https://crash-stats.mozilla.com/report/index/db39437d-2d42-4e58-9d33-a13e90181116

is the same crash (which it looks like), I have a recreation scenario on Windows.

Create a new profile and make sure it doesn't show the "multiple tabs open" message on shutdown.

1. Use the web page method to install an add-on which includes a search engine (eg. https://ext.ask.com/index.jhtml)
2. When the second prompt is showing, which is for the default search, close Firefox using the X button in the window title bar

If anything interferes with the closing of Firefox, it won't happen (dialog popup).

What you'll see is that multiple processes are left open in the task manager and after a minute or so, Firefox crashes.
(In reply to Mike Kaply [:mkaply] from comment #45)
> Assuming this:
> 
> https://crash-stats.mozilla.com/report/index/db39437d-2d42-4e58-9d33-
> a13e90181116
> 
> is the same crash (which it looks like), I have a recreation scenario on
> Windows.
> 
> Create a new profile and make sure it doesn't show the "multiple tabs open"
> message on shutdown.
> 
> 1. Use the web page method to install an add-on which includes a search
> engine (eg. https://ext.ask.com/index.jhtml)

That is bug 1493770, which is assigned to you.
See Also: → 1509352
Duplicate of this bug: 1509352
Depends on: 1525729
Crash Signature: AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org,Extension shutdown: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} ] → AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org] [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: fxmonitor@mozilla.org,Extension shutdown: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} ] [@ As…
Duplicate of this bug: 1535311

Rob, are the the remaining crashes on target to get a fix for 67?

Flags: needinfo?(rob)

No, the others are not in 67.

The top cause (asyncEmitManifestEntry(\"chrome_settings_overrides\")) was fixed by bug 1525729, just before the weekend, in time for Firefox 67.

The second top cause (asyncEmitManifestEntry(\"background\")) was supposedly addressed by bug 1501375, but there are still crash reports with the signature.

The others haven't been addressed yet. Each of the others (from comment 44) should be tracked separately, but I'm keeping them in this one bug report since the crash signatures are indistinguishable (the relevant information is in the async shutdown timeout field, which is not recognized by BMO).

Flags: needinfo?(rob)
Depends on: 1542181
Depends on: 1543354

I suspect that many of these crashes occur when the lengthy extension startup continues after shutdown has already been requested, as 0.4% of the startups reported by telemetry take over 50 seconds (0.4% on Beta 67, 0.5% on Nightly 68).
Adding a few checks and early returns from the extension starup could reduce the number of timeouts, but there will always be some timeouts under unfortunate circumstances on constrained systems (e.g. slow file access).

The page_action handler is a clear example that couldn't have been the cause of a timeout. It awaits StartupCache.get, but at that point the startup cache has already been read (to read extension manifests), so the call is expected to resolve within a few microtasks. Yet there were 10 AsyncShutdown timeout crashes associated with page_action in the past three months, 9 of these are from one of the built-in extensions (screenshots, webcompat).

I filed two new dependencies (bug 1543354 and bug 1542181).
Bug 1543354 is actionable, whereas bug 1542181 is just a report of the issue with no clear path forwards, as it is likely dependent on file IO.
The crash rates for the remaining signatures could be lowered by adding early returns to the Extension's startup if shutdown has started.


For comparison, update to comment 44:

1. 1272 at asyncEmitManifestEntry(\"chrome_settings_overrides\") (1 after bug 1525729 landed)
2. 499 at asyncEmitManifestEntry(\"background\") - bug 1543354
3. 933 at at "Startup: Run manifest\""
4. 27 at "Startup: Loading manifest\""
5. 30 at "Shutdown: Flushed jar cache"
6. 21 at "Shutdown: Emit shutdown"
7. 15 at asyncEmitManifestEntry(\"commands\")
8. 351 at "Shutdown: Storage" (significant regression - see bug 1542181)
9. 2 at asyncEmitManifestEntry(\"chrome_url_overrides\")
10. 0 at asyncEmitManifestEntry(\"browser_action\") (7 in past three months)
11. 0 at "Shutdown: Flush jar cache" (2 in past three months)
12. 0 at asyncEmitManifestEntry(\"page_action\") (10 in past three months) (NEW)

  1. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry(%5C%5C%5C%22chrome_settings_overrides%5C%5C%5C%22)&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  2. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22background%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  3. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Run%20manifest%5C%22&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  4. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Startup%3A%20Loading%20manifest%5C%22&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  5. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flushed%20jar%20cache&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  6. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Emit%20shutdown&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  7. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22commands%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  8. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Storage&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  9. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22chrome_url_overrides%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  10. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22browser_action%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  11. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~Shutdown%3A%20Flush%20jar%20cache&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
  12. https://crash-stats.mozilla.com/search/?signature=~AsyncShutdownTimeout%20%7C%20profile-change-teardown%20%7C%20Extension%20shutdown%3A&async_shutdown_timeout=~asyncEmitManifestEntry%28%5C%5C%5C%22page_action%5C%5C%5C%22%29&date=%3E%3D2019-04-03T00%3A00%3A00.000Z&date=%3C2019-04-10T00%3A00%3A00.000Z&_facets=signature&_facets=version&page=1&_sort=-date&_columns=date&_columns=version&_columns=build_id&_columns=async_shutdown_timeout#crash-reports
Depends on: 1543384
Duplicate of this bug: 1559448
Attachment #9010687 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.