1681745 - Startup Crash in [@ mozilla::ErrorLoadingSheet]

Status: NEW

(Firefox :: Installer, defect)

Description

[Camelia Badau [:cbadau], Desktop QA]

Crash report: https://crash-stats.mozilla.org/report/index/74d718ab-c6a3-4599-b769-9bd7c0201210

MOZ_CRASH Reason: LoadSheetSync failed with error 80520012 loading built-in stylesheet 'chrome://global/content/xul.css'

Top 10 frames of crashing thread:

0 xul.dll mozilla::ErrorLoadingSheet layout/style/GlobalStyleSheetCache.cpp:530
1 xul.dll mozilla::GlobalStyleSheetCache::LoadSheet layout/style/GlobalStyleSheetCache.cpp:565
2 xul.dll mozilla::GlobalStyleSheetCache::LoadSheetURL layout/style/GlobalStyleSheetCache.cpp:510
3 xul.dll mozilla::GlobalStyleSheetCache::XULSheet layout/style/UserAgentStyleSheetList.h:40
4 xul.dll mozilla::GlobalStyleSheetCache::GlobalStyleSheetCache layout/style/GlobalStyleSheetCache.cpp:239
5 xul.dll static mozilla::GlobalStyleSheetCache::Singleton layout/style/GlobalStyleSheetCache.cpp:460
6 xul.dll mozilla::dom::Document::FillStyleSetUserAndUASheets dom/base/Document.cpp:2766
7 xul.dll mozilla::dom::Document::CreatePresShell dom/base/Document.cpp:6150
8 xul.dll nsDocumentViewer::InitPresentationStuff layout/base/nsDocumentViewer.cpp:768
9 xul.dll nsDocumentViewer::InitInternal layout/base/nsDocumentViewer.cpp:974

I've encountered this startup crash several times on Windows 10 x64, using Firefox 78.6.0 ESR, latest Nightly 85.0a1 and Firefox 84 RC.

I've tested on machines with Cisco AMP for Endpoints (7.2.11.11804) - our default antivirus on Softvision machines, and this antivirus seems to detect the omni.ja file as a corrupt file and blocks it.

On machines with other antiviruses (Bitdefender, Windows Defender), everything looks good and no crash occur.

Suggested severity: S1

Comment 1

[Emilio Cobos Álvarez (:emilio)]

This is NS_ERROR_FILE_NOT_FOUND. We can't really proceed without fetching the stylesheets. Seems like an issue with the antivirus, what's the usual procedure for this?

Comment 2

[Gabriele Svelto [:gsvelto]]

If the bug is caused by an injected DLL we usually follow this procedure. That doesn't seem to be the case here though, is the AV actually blocking the file read? I don't think we can do much about it. That being said I suspect there's something else going wrong here since there's quite a few recent crash reports coming from Android and I doubt that it's an AV problem there.

Comment 3

[Marco Castelluccio [:marco]]

(In reply to Gabriele Svelto [:gsvelto] from comment #2)

If the bug is caused by an injected DLL we usually follow this procedure. That doesn't seem to be the case here though, is the AV actually blocking the file read? I don't think we can do much about it. That being said I suspect there's something else going wrong here since there's quite a few recent crash reports coming from Android and I doubt that it's an AV problem there.

I think I remember similar problems caused by some AV in the past. Maybe the Android crashes have the same signature but are not the same crash? After all, this crash can happen any time there is a corrupted installation, and corruptions can have different causes.
I guess we should try to contact Cisco to report this problem.

Comment 4

[Romain Testard [:RT]]

I'm asking people at Cisco to help find someone who can help there - sounds like we have evidence that AMP is the root cause per https://bugzilla.mozilla.org/show_bug.cgi?id=1681745#c0

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Camelia, please provide further information:

• Did this issue affect only one machine or multiple?
• What are the exact builds (version + build ID) which are affected?
• Can you share the omni.ja files it complains about? They are too big to get attached here, e.g. use Google Drive.

Comment 6

[Camelia Badau [:cbadau], Desktop QA]

Initially, the problem was reproduced on several machines with Cisco AMP, but today we retested and we have only one machine on which the problem still reproduces. Here, on this machine, the problem is reproduced only on the old ESR78.6.0 build installed on Dec 10, buildID: 20201207224150 (here is the omni.ja file for this build).
For new installs of ESR78.6.0 (or any other build: Release, Nightly) on this machine, everything works ok and the Firefox opens correctly.

On other machines, we deleted all Mozilla folders from AppData (Local, LocalLow and Roaming), and so the problem does not recur anymore.

Comment 7

[ivberlin]

Hi Camelia,
I'm a Cisco TSA covering AMP in Europe. I have not been able to reproduce your issue neither with the latest connector version (7.3.5) neither with the one you are currently using (7.2.11).
As you may know there are multiple prevention engines inside AMP. Would you accept i contact you directly so you could share with me more details about it? (the detailed event in the AMP console would be more than welcome).
Thank you
Ivan

Comment 8

[Camelia Badau [:cbadau], Desktop QA]

(In reply to ivberlin from comment #7)

Hi Camelia,
I'm a Cisco TSA covering AMP in Europe. I have not been able to reproduce your issue neither with the latest connector version (7.3.5) neither with the one you are currently using (7.2.11).
As you may know there are multiple prevention engines inside AMP. Would you accept i contact you directly so you could share with me more details about it? (the detailed event in the AMP console would be more than welcome).
Thank you
Ivan

Hi Ivan,

Sure, you can contact me directly on [e-mail].

Comment 9

[Wayne Mery (:wsmwk)]

Also startup crash for Thunderbird 78. bp-c744e5bf-1993-41b3-8325-a960b0210319

Comment 10

[[e-mail]]

I am running into this issue on a vmWare Horizon virtual machine on Win10 20H2.

What tends to happen is that I launch Firefox during login and it starts up just fine. Then if Firefox crashes or I accidentally close it during the day, it refuses to launch again and gives the below error. It won't even launch in Safe Mode (firefox.exe -safe-mode). The only option is to log-off from the virtual machine, wait for my session to be destroyed and re-login. Our vmWare administrators use something called AppSense to sync the necessary files. I also lack administrative rights on the virtual machine.

I don't see any security software (McAfee, Norton) or Cisco AMP but there is Seclore Endpoint.

crash.main.3
1628933812
c624822c - 59bb - 416b - 996d - 4e58395e491a{
"BuildID" : "20210706194121",
"CPUMicrocodeVersion" : "0x2000065",
"DOMIPCEnabled" : "1",
"InstallTime" : "1626327210",
"ProductID" : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
"ProductName" : "Firefox",
"ReleaseChannel" : "esr",
"SafeMode" : "0",
"ServerURL" : "https://crash-reports.mozilla.com/submit?id={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&version=78.12.0&buildid=20210706194121",
"StartupCrash" : "1",
"StartupTime" : "1628933811",
"Vendor" : "Mozilla",
"Version" : "78.12.0",
"CrashTime" : "1628933812",
"UptimeTS" : "16.24939",
"SecondsSinceLastCrash" : "102",
"BreakpadReserveAddress" : "2552009785344",
"BreakpadReserveSize" : "83886080",
"MozCrashReason" : "LoadSheetSync failed with error 80520012 loading built-in stylesheet 'chrome://global/content/xul.css'",
"ThreadIdNameMapping" : "10732:Gecko_IOThread,2116:Timer,14000:Link Monitor #1,11296:Socket Thread,13932:Permission,9320:JS Watchdog,13088:JS Helper,12124:JS Helper,10668:JS Helper,13480:BackgroundThreadPool #1,13676:IPDL Background,"
}

Comment 11

[Wayne Mery (:wsmwk)]

There is a recent sharp and significant increase - 4x overall - so now Firefox 102.4.0esr and 102.5.0esr this ranks #21. #49 for Thunderbird 102.5.0. 99% startup crashes.

Comment 12

[Haik Aftandilian [:haik]]

The majority of the recent crash reports I looked at had a Trend Micro module loaded, although not all of them.

@Greg, could you take a look at this one? This might be a case of many different third-party issues causing the same crash.

Comment 13

[Greg Stoll :gstoll]

Yeah, I'll take a look!

Comment 14

[Greg Stoll :gstoll]

The spike from a few months ago seems to be only in the ESR channel - we still see crashes in release, etc., but those appear steady over time.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on release (startup)

:gstoll, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 16

[Greg Stoll :gstoll]

The most recent spike in these crashes is in the release branch, and it corresponds with the release date of 113.0...but 75% of the crashes are from 112.0.2. I wonder if this indicates that people with 112.0.2 were trying to update to 113.0, the update failed somehow but left files in a bad state, and then launching the old version led to this crash?

Passing this on to Installers to see if this is plausible.

Comment 17

[Wayne Mery (:wsmwk)]

Crash rate dropped in 113. Probably no longer a topcrash.
https://crash-stats.mozilla.org/signature/?signature=mozilla::ErrorLoadingSheet&date=>=2023-05-28T00:42:00.000Z&date=<2023-06-04T00:42:00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit BugBot documentation.