1980212 - Crash in [@ AsyncShutdownTimeout | profile-change-teardown | Extension shutdown: {d634138d-c276-4fc8-924b-40a0ea21d284}]

Status: RESOLVED WORKSFORME

(WebExtensions :: Developer Outreach, defect)

Description

[Andrew McCreight [:mccr8]]

Crash report: https://crash-stats.mozilla.org/report/index/5b06d682-5706-4bcb-ad49-4676c0250729

MOZ_CRASH Reason:

[Parent 4211, Main Thread] ###!!! ABORT: file resource://gre/modules/addons/XPIProvider.sys.mjs:2880

Top 10 frames:

0  XUL  MOZ_CrashSequence(void*, long)  mfbt/Assertions.h:248
0  XUL  MOZ_Crash(char const*, int, char const*)  mfbt/Assertions.h:381
0  XUL  Abort(char const*)  xpcom/base/nsDebugImpl.cpp:528
0  XUL  NS_DebugBreak  xpcom/base/nsDebugImpl.cpp:485
1  XUL  nsDebugImpl::Abort(char const*, int)  xpcom/base/nsDebugImpl.cpp:129
2  XUL  NS_InvokeByIndex
3  XUL  CallMethodHelper::Invoke()  js/xpconnect/src/XPCWrappedNative.cpp:1620
3  XUL  CallMethodHelper::Call()  js/xpconnect/src/XPCWrappedNative.cpp:1174
3  XUL  XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode)  js/xpconnect/src/XPCWrappedNative.cpp:1120
4  XUL  XPC_WN_CallMethod(JSContext*, unsigned int, JS::Value*)  js/xpconnect/src/XPCWrappedNativeJSOps.cpp:966

The crash spike reporter reported an increase in crashes with this signature. The extension in question is 1Password, so I expect this is another symptom of bug 1980009, but I figured I'd file it so we have a record of it.

Comment 1

[Rob Wu [:robwu]]

In the annotations tab of the crash report, there is more information than the generic AsyncShutdownTimeout crash. The AsyncShutdownTimeout field of the crash report in the link has the following content:

{"phase":"profile-change-teardown","conditions":[{"name":"Extension shutdown: {d634138d-c276-4fc8-924b-40a0ea21d284}","state":{"state":"Startup: Run manifest, asyncEmitManifestEntry(\"background\")"},"filename":"resource://gre/modules/addons/XPIProvider.sys.mjs","lineNumber":2880,"stack":["resource://gre/modules/addons/XPIProvider.sys.mjs:startup/<:2880","resource://gre/modules/AsyncShutdown.sys.mjs:observe:569"]}]}

This signature is an indication that the extension's background page tried to start up at browser startup, but then gets stuck (bug 1543354).

In bug 1980009 we identified a situation where the 1Password extension enters an infinite loop. That could trigger this bug. It can also affect other extensions, if these other extensions are not getting the CPU time to start up themselves, but that would be associated with a different signature.

Comment 2

[Pascal Chevrel:pascalc]

Tracking because of the recent spike in case we could have a fix/mitigation for next week dot release.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

:mccr8, since this bug is a regression, could you fill (if possible) the regressed_by field?
For more information, please visit BugBot documentation.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 5 desktop browser crashes on Mac on release

For more information, please visit BugBot documentation.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox141 (release). We have limited time to fix this, the soft freeze is in 14 days. However, the bug still isn't assigned.

:mixedpuppy, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 6

[Andrew McCreight [:mccr8]]

This is not a Firefox regression, as far as I am aware.

Comment 7

[Rob Wu [:robwu]]

I'll self-assign since I am actively involved in the investigation of this issue. The most likely outcome is for this bug to be closed once the 1Password extension is updated.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Can we close this now?

Comment 9

[Andrew McCreight [:mccr8]]

It looks like this is still happening, so maybe it is a separate issue. For instance, this crash happened today: bp-1251bd49-d30e-4662-ba12-eecab0250805, and they are shown as having version 8.11.4.27, which is the version in bug 1980009 that is supposed to contain the fix.

Comment 10

[Rob Wu [:robwu]]

(In reply to Andrew McCreight [:mccr8] from comment #9)

It looks like this is still happening, so maybe it is a separate issue.

The crash spike trends downwards, which is promising. Even after the update has been published, I expect crashes like these to still linger until users have updated.

Even when a user updates, they may still be affected by bug 1980009, but it resolves on its own after a restart.

For instance, this crash happened today: bp-1251bd49-d30e-4662-ba12-eecab0250805, and they are shown as having version 8.11.4.27, which is the version in bug 1980009 that is supposed to contain the fix.

In the Telemetry environment tab, I see (excerpt below) that updateDay is 20305, which is today according to new Date(20305 * 24 * 36e5) = 5 august.

Even if you find crashes with updateDay before today, it is possible that the Firefox instance was still running before the update.

We should only be worried if we see a significant number of users with this AsyncShutdown crash where updateDay is before the current uptime. I checked the 10 of of the most recent crashes out with an uptime of a bit over 2 days, opened the Telemetry Environment tab for each, and searched for d634138d-c276-4fc8-924b-40a0ea21d284}:. Most have updateDay 20305, but three had updateDay 20304. Of these, two have an uptime that puts the startup to yesterday (example and example). One other had an uptime of 7 hours, but its crash time was yesterday.

{d634138d-c276-4fc8-924b-40a0ea21d284}: {
    appDisabled: false,
    blocklisted: false,
    description: "The best way to experience 1Password in your browser. Easily sign in to sites, generate passwords, a",
    foreignInstall: false,
    hasBinaryComponents: false,
    installDay: 20097,
    isSystem: false,
    isWebExtension: true,
    multiprocessCompatible: true,
    name: "1Password – Password Manager",
    scope: 1,
    signedState: 2,
    type: "extension",
    updateDay: 20305,
    userDisabled: false,
    version: "8.11.4.27"
}

Comment 11

[Rob Wu [:robwu]]

The crash stats are not sharply spiking down.

I want to create a crashstats query (or a script that uses the crashstats API) to retrieve all crash reports, and compare the following:

• uptime
• updateDay from telemetry environment for the 1Password add-on.
• crash time (different from submission time!)

... and see if there are any anomalies. I have spot-checked crashes every day since the update was published, and so far the data does not rule out the possibility that these crashes are due to an old 1Password still being part of the profile (and that the update happened in the same session).

Comment 12

[Rob Wu [:robwu]]

The crash spike dropped significantly over the past few days, but that includes the last two days (weekend). I'll continue monitoring for a few days and close this bug if it flattens.

Comment 13

[Rob Wu [:robwu]]

The spike is almost flattening; in the past 24 hours there have only been 21 crashes. In my final analysis below, I can still attribute all reported crashes to the buggy 1Password version. Since the problem is well-understood, and there are no actionable tasks left, I'll close this bug.

I looked separately at crashes with uptime < 20000s (a bit over 2 days), of which there are 10 - all of them have received a 1Password update on the day of the crash. The crash reflects the previous version of the extension.

The other 11 crashes are longer sessions. 3 of them are up to date by now, but the crash is still associated with the previous version.

The remaining 8 crashes with long sessions still have the buggy 1Password version installed (8.11.2.21). It is possible that these users disabled auto-update of extensions. Interestingly, the session duration indicates that these user did not experience the degradation of an unresponsive extension process (at 100% CPU). Most of these users do not have extensions that broadly suspends all network requests (e.g. ad blockers), which limits the user-perceived impact.

• bp-029d3bbb-39e1-44d3-a838-3be590250812
• bp-77669a75-e5fb-4454-8454-e371d0250812
• bp-a7a32bec-3523-4de3-8da7-2f27e0250812
• bp-d23772a2-39fe-4272-aee5-4c2230250812
• bp-2bff430d-e863-415d-949f-297260250812
• bp-d6690b5e-3417-4e99-81e9-8ebb40250812
• bp-7a3ff3c1-e9aa-4004-aefd-fdf320250811
• bp-6893f8ca-ed6e-4de4-8eee-1bcfb0250811