1782924 - Crash in [@ AsyncShutdownTimeout | IOUtils: waiting for profileBeforeChange IO to complete | JSON store: writing data]

Status: RESOLVED FIXED

(Toolkit Graveyard :: OS.File, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

~60% of crashes in first 5 minutes after application launch. 20-50 crash reports per 104 beta version, almost only on Windows. Not sure if this is a signature change - couldn't find a signature with similar volume for 103 beta versions. Barret, could you investigate this?

Crash report: https://crash-stats.mozilla.org/report/index/0b16cc95-db90-49d4-9dbd-7f33f0220802

MOZ_CRASH Reason: [Parent 4472, Main Thread] ###!!! ABORT: file resource://gre/modules/JSONFile.jsm:136

  this._finalizeAt.addBlocker(
    "JSON store: writing data",
    this._finalizeInternalBound
  );

Top 10 frames of crashing thread:

0 xul.dll NS_DebugBreak xpcom/base/nsDebugImpl.cpp:496
1 xul.dll nsDebugImpl::Abort xpcom/base/nsDebugImpl.cpp:129
2 xul.dll XPTC__InvokebyIndex 
3 xul.dll static XPCWrappedNative::CallMethod js/xpconnect/src/XPCWrappedNative.cpp:1125
4 xul.dll XPC_WN_CallMethod js/xpconnect/src/XPCWrappedNativeJSOps.cpp:965
5 xul.dll Interpret js/src/vm/Interpreter.cpp:3313
6 xul.dll js::Call js/src/vm/Interpreter.cpp:602
7 xul.dll PromiseReactionJob js/src/builtin/Promise.cpp:2241
8 xul.dll js::Call js/src/vm/Interpreter.cpp:602
9 xul.dll mozilla::PromiseJobRunnable::Run xpcom/base/CycleCollectedJSContext.cpp:213

Comment 1

[Barret Rennie [:barret] (they/them)]

There are several other crashes that include JSONFile.jsm, but they usually have other things in the log besides itself (like crashmonitor or sessionstore).

It would be nice if JSONFile.jsm included the (sanitized) file it was writing in its blocker information. I've filed bug 1782990 so we can include that information and then follow up addressing the root cause.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox104 (beta). We have limited time to fix this, the soft freeze is in 14 days. However, the bug still isn't assigned.

:gcp, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Comment 3

[Barret Rennie [:barret] (they/them)]

Looking at the crash report, this might be related to :nalexander's backgroundtask work

:nalexander, the crash report contains the following AsyncShutdownTimeout annotation:

{
    "phase": "IOUtils: waiting for profileBeforeChange IO to complete",
    "conditions": [
        {
            "name": "JSON store: writing data",
            "state": "(none)",
            "filename": "resource://gre/modules/JSONFile.jsm",
            "lineNumber": 136,
            "stack": [
                "resource://gre/modules/JSONFile.jsm:JSONFile:136",
                "resource://gre/modules/BackgroundUpdate.jsm:scheduleFirefoxMessagingSystemTargetingSnapshotting:630",
                "resource:///modules/BrowserGlue.jsm:task:2810",
                "resource:///modules/BrowserGlue.jsm:_scheduleStartupIdleTasks/<:2888"
            ]
        }
    ]
}

Comment 4

[Nick Alexander :nalexander [he/him]]

Mmm, this is unfortunate. Short run-times mean that we're requesting all of the targeting state at shutdown time (intentionally), but that appears to be timing out, possibly since some of those components are shut down or shutting down. Leaving NI since I'll need to address this in some manner.

Comment 5

[Hsin-Yi Tsai (she/her) [:hsinyi]]

As a reminder we only have two betas left this week to fix this crash. Thank you.

Comment 6

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1782924 - Avoid crash writing Firefox Messaging System targeting information at shutdown. r?barret

I tested this as much as I could locally -- hence the dump
workaround -- but I have no particular suggestion for how to test this
in automation. Even triggering the targeting snapshotting during
shutdown requires the timers and shutdown process to line up in a way
that's not trivial to guarantee.

Comment 7

[Nick Alexander :nalexander [he/him]]

Comment on attachment 9289319 [details]
Bug 1782924 - Avoid crash writing Firefox Messaging System targeting information at shutdown. r?barret

Beta/Release Uplift Approval Request

• User impact if declined: Low volume of shutdown crashes.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's hard to imagine that this patch would increase the volume of shutdown crashes, since it avoids a potentially expensive and time-consuming operation if shutdown is initiated.
• String changes made/needed:
• Is Android affected?: Yes

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6824b40225d8
Avoid crash writing Firefox Messaging System targeting information at shutdown. r=application-update-reviewers,bytesized

Comment 9

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/6824b40225d8

Comment 10

[Dianna Smith [:diannaS]]

Comment on attachment 9289319 [details]
Bug 1782924 - Avoid crash writing Firefox Messaging System targeting information at shutdown. r?barret

Approved for 104.0b9

Comment 11

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/089006095e3b