1681745 - Startup Crash in [@ mozilla::ErrorLoadingSheet]

Status: NEW

(Firefox :: Installer, defect)

Description

[Camelia Badau [:cbadau], Desktop Test Engineering]

Crash report: https://crash-stats.mozilla.org/report/index/74d718ab-c6a3-4599-b769-9bd7c0201210

MOZ_CRASH Reason: LoadSheetSync failed with error 80520012 loading built-in stylesheet 'chrome://global/content/xul.css'

Top 10 frames of crashing thread:

0 xul.dll mozilla::ErrorLoadingSheet layout/style/GlobalStyleSheetCache.cpp:530
1 xul.dll mozilla::GlobalStyleSheetCache::LoadSheet layout/style/GlobalStyleSheetCache.cpp:565
2 xul.dll mozilla::GlobalStyleSheetCache::LoadSheetURL layout/style/GlobalStyleSheetCache.cpp:510
3 xul.dll mozilla::GlobalStyleSheetCache::XULSheet layout/style/UserAgentStyleSheetList.h:40
4 xul.dll mozilla::GlobalStyleSheetCache::GlobalStyleSheetCache layout/style/GlobalStyleSheetCache.cpp:239
5 xul.dll static mozilla::GlobalStyleSheetCache::Singleton layout/style/GlobalStyleSheetCache.cpp:460
6 xul.dll mozilla::dom::Document::FillStyleSetUserAndUASheets dom/base/Document.cpp:2766
7 xul.dll mozilla::dom::Document::CreatePresShell dom/base/Document.cpp:6150
8 xul.dll nsDocumentViewer::InitPresentationStuff layout/base/nsDocumentViewer.cpp:768
9 xul.dll nsDocumentViewer::InitInternal layout/base/nsDocumentViewer.cpp:974

I've encountered this startup crash several times on Windows 10 x64, using Firefox 78.6.0 ESR, latest Nightly 85.0a1 and Firefox 84 RC.

I've tested on machines with Cisco AMP for Endpoints (7.2.11.11804) - our default antivirus on Softvision machines, and this antivirus seems to detect the omni.ja file as a corrupt file and blocks it.

On machines with other antiviruses (Bitdefender, Windows Defender), everything looks good and no crash occur.

Suggested severity: S1

Comment 1

[Emilio Cobos Álvarez (:emilio)]

This is NS_ERROR_FILE_NOT_FOUND. We can't really proceed without fetching the stylesheets. Seems like an issue with the antivirus, what's the usual procedure for this?

Comment 2

[Gabriele Svelto [:gsvelto]]

If the bug is caused by an injected DLL we usually follow this procedure. That doesn't seem to be the case here though, is the AV actually blocking the file read? I don't think we can do much about it. That being said I suspect there's something else going wrong here since there's quite a few recent crash reports coming from Android and I doubt that it's an AV problem there.

Comment 3

[Marco Castelluccio [:marco]]

(In reply to Gabriele Svelto [:gsvelto] from comment #2)

If the bug is caused by an injected DLL we usually follow this procedure. That doesn't seem to be the case here though, is the AV actually blocking the file read? I don't think we can do much about it. That being said I suspect there's something else going wrong here since there's quite a few recent crash reports coming from Android and I doubt that it's an AV problem there.

I think I remember similar problems caused by some AV in the past. Maybe the Android crashes have the same signature but are not the same crash? After all, this crash can happen any time there is a corrupted installation, and corruptions can have different causes.
I guess we should try to contact Cisco to report this problem.

Comment 4

[Romain Testard [:RT]]

I'm asking people at Cisco to help find someone who can help there - sounds like we have evidence that AMP is the root cause per https://bugzilla.mozilla.org/show_bug.cgi?id=1681745#c0

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Camelia, please provide further information:

• Did this issue affect only one machine or multiple?
• What are the exact builds (version + build ID) which are affected?
• Can you share the omni.ja files it complains about? They are too big to get attached here, e.g. use Google Drive.

Comment 6

[Camelia Badau [:cbadau], Desktop Test Engineering]

Initially, the problem was reproduced on several machines with Cisco AMP, but today we retested and we have only one machine on which the problem still reproduces. Here, on this machine, the problem is reproduced only on the old ESR78.6.0 build installed on Dec 10, buildID: 20201207224150 (here is the omni.ja file for this build).
For new installs of ESR78.6.0 (or any other build: Release, Nightly) on this machine, everything works ok and the Firefox opens correctly.

On other machines, we deleted all Mozilla folders from AppData (Local, LocalLow and Roaming), and so the problem does not recur anymore.

Comment 7

[ivberlin]

Hi Camelia,
I'm a Cisco TSA covering AMP in Europe. I have not been able to reproduce your issue neither with the latest connector version (7.3.5) neither with the one you are currently using (7.2.11).
As you may know there are multiple prevention engines inside AMP. Would you accept i contact you directly so you could share with me more details about it? (the detailed event in the AMP console would be more than welcome).
Thank you
Ivan

Comment 8

[Camelia Badau [:cbadau], Desktop Test Engineering]

(In reply to ivberlin from comment #7)

Hi Camelia,
I'm a Cisco TSA covering AMP in Europe. I have not been able to reproduce your issue neither with the latest connector version (7.3.5) neither with the one you are currently using (7.2.11).
As you may know there are multiple prevention engines inside AMP. Would you accept i contact you directly so you could share with me more details about it? (the detailed event in the AMP console would be more than welcome).
Thank you
Ivan

Hi Ivan,

Sure, you can contact me directly on [e-mail].

Comment 9

[Wayne Mery (:wsmwk)]

Also startup crash for Thunderbird 78. bp-c744e5bf-1993-41b3-8325-a960b0210319

Comment 10

[[e-mail]]

I am running into this issue on a vmWare Horizon virtual machine on Win10 20H2.

What tends to happen is that I launch Firefox during login and it starts up just fine. Then if Firefox crashes or I accidentally close it during the day, it refuses to launch again and gives the below error. It won't even launch in Safe Mode (firefox.exe -safe-mode). The only option is to log-off from the virtual machine, wait for my session to be destroyed and re-login. Our vmWare administrators use something called AppSense to sync the necessary files. I also lack administrative rights on the virtual machine.

I don't see any security software (McAfee, Norton) or Cisco AMP but there is Seclore Endpoint.

crash.main.3
1628933812
c624822c - 59bb - 416b - 996d - 4e58395e491a{
"BuildID" : "20210706194121",
"CPUMicrocodeVersion" : "0x2000065",
"DOMIPCEnabled" : "1",
"InstallTime" : "1626327210",
"ProductID" : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
"ProductName" : "Firefox",
"ReleaseChannel" : "esr",
"SafeMode" : "0",
"ServerURL" : "https://crash-reports.mozilla.com/submit?id={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&version=78.12.0&buildid=20210706194121",
"StartupCrash" : "1",
"StartupTime" : "1628933811",
"Vendor" : "Mozilla",
"Version" : "78.12.0",
"CrashTime" : "1628933812",
"UptimeTS" : "16.24939",
"SecondsSinceLastCrash" : "102",
"BreakpadReserveAddress" : "2552009785344",
"BreakpadReserveSize" : "83886080",
"MozCrashReason" : "LoadSheetSync failed with error 80520012 loading built-in stylesheet 'chrome://global/content/xul.css'",
"ThreadIdNameMapping" : "10732:Gecko_IOThread,2116:Timer,14000:Link Monitor #1,11296:Socket Thread,13932:Permission,9320:JS Watchdog,13088:JS Helper,12124:JS Helper,10668:JS Helper,13480:BackgroundThreadPool #1,13676:IPDL Background,"
}

Comment 11

[Wayne Mery (:wsmwk)]

There is a recent sharp and significant increase - 4x overall - so now Firefox 102.4.0esr and 102.5.0esr this ranks #21. #49 for Thunderbird 102.5.0. 99% startup crashes.

Comment 12

[Haik Aftandilian [:haik]]

The majority of the recent crash reports I looked at had a Trend Micro module loaded, although not all of them.

@Greg, could you take a look at this one? This might be a case of many different third-party issues causing the same crash.

Comment 13

[Greg Stoll :gstoll]

Yeah, I'll take a look!

Comment 14

[Greg Stoll :gstoll]

The spike from a few months ago seems to be only in the ESR channel - we still see crashes in release, etc., but those appear steady over time.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on release (startup)

:gstoll, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 16

[Greg Stoll :gstoll]

The most recent spike in these crashes is in the release branch, and it corresponds with the release date of 113.0...but 75% of the crashes are from 112.0.2. I wonder if this indicates that people with 112.0.2 were trying to update to 113.0, the update failed somehow but left files in a bad state, and then launching the old version led to this crash?

Passing this on to Installers to see if this is plausible.

Comment 17

[Wayne Mery (:wsmwk)]

Crash rate dropped in 113. Probably no longer a topcrash.
https://crash-stats.mozilla.org/signature/?signature=mozilla::ErrorLoadingSheet&date=>=2023-05-28T00:42:00.000Z&date=<2023-06-04T00:42:00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit BugBot documentation.

Comment 19

[Gabriele Svelto [:gsvelto]]

There is a very large spike here of Windows 10 users with the following crash reason:

LoadSheetSync failed with error 80520012 loading built-in stylesheet 'resource://gre/res/designmode.css'

80520012 is NS_ERROR_FILE_NOT_FOUND and it's a startup crash, so the users are livid in the comments. The crash seems to be affecting only users on version 133.0.3. There's only a handful of crashes on file for version 134.0 and only one of them is for Windows 10 but the cause is different, so this might not be affecting people on 134.0. However I wonder if those experiencing the crash can upgrade or are stuck with the broken version.

Andrew what do you think? Is it worth to open an incident for this?

Comment 20

[Andrew McCreight [:mccr8]]

(In reply to Gabriele Svelto [:gsvelto] f

Andrew what do you think? Is it worth to open an incident for this?

I'm not sure, though it does look bad. I'd ask in #release-coordination on Slack. They could at least stop the update. I did see a few comments talking about how Firefox wouldn't start since they updated. I don't know how we'll make any progress on understanding the problem here.

Comment 21

[Andrew McCreight [:mccr8]]

I brought this up on Slack and there is some discussion there. Ryanvm mentioned bug 1388910, which deleted designmode.css. I found one crash that actually had a telemetry environment and it has the value 134.0 for displayVersion but 133.0.3 for version and platformVersion.

Comment 22

[Andrew McCreight [:mccr8]]

Looking at how the telemetry values are generated, I think displayVersion comes from the omnija (via AppConstants) and version and platformVersion come from the executable (via nsXULAppInfo), so that would be consistent with the mixed installation theory.

Comment 23

[Andrew McCreight [:mccr8]]

I split off a fresh bug for this crash spike to reduce confusion, bug 1941134.