1681745 - Startup Crash in [@ mozilla::ErrorLoadingSheet]

Status: RESOLVED FIXED

(Firefox :: Installer, defect, P1)

Description

[Camelia Badau [:cbadau], Desktop Test Engineering]

Crash report: https://crash-stats.mozilla.org/report/index/74d718ab-c6a3-4599-b769-9bd7c0201210

MOZ_CRASH Reason: LoadSheetSync failed with error 80520012 loading built-in stylesheet 'chrome://global/content/xul.css'

Top 10 frames of crashing thread:

0 xul.dll mozilla::ErrorLoadingSheet layout/style/GlobalStyleSheetCache.cpp:530
1 xul.dll mozilla::GlobalStyleSheetCache::LoadSheet layout/style/GlobalStyleSheetCache.cpp:565
2 xul.dll mozilla::GlobalStyleSheetCache::LoadSheetURL layout/style/GlobalStyleSheetCache.cpp:510
3 xul.dll mozilla::GlobalStyleSheetCache::XULSheet layout/style/UserAgentStyleSheetList.h:40
4 xul.dll mozilla::GlobalStyleSheetCache::GlobalStyleSheetCache layout/style/GlobalStyleSheetCache.cpp:239
5 xul.dll static mozilla::GlobalStyleSheetCache::Singleton layout/style/GlobalStyleSheetCache.cpp:460
6 xul.dll mozilla::dom::Document::FillStyleSetUserAndUASheets dom/base/Document.cpp:2766
7 xul.dll mozilla::dom::Document::CreatePresShell dom/base/Document.cpp:6150
8 xul.dll nsDocumentViewer::InitPresentationStuff layout/base/nsDocumentViewer.cpp:768
9 xul.dll nsDocumentViewer::InitInternal layout/base/nsDocumentViewer.cpp:974

I've encountered this startup crash several times on Windows 10 x64, using Firefox 78.6.0 ESR, latest Nightly 85.0a1 and Firefox 84 RC.

I've tested on machines with Cisco AMP for Endpoints (7.2.11.11804) - our default antivirus on Softvision machines, and this antivirus seems to detect the omni.ja file as a corrupt file and blocks it.

On machines with other antiviruses (Bitdefender, Windows Defender), everything looks good and no crash occur.

Suggested severity: S1

Comment 1

[Emilio Cobos Álvarez [:emilio]]

This is NS_ERROR_FILE_NOT_FOUND. We can't really proceed without fetching the stylesheets. Seems like an issue with the antivirus, what's the usual procedure for this?

Comment 2

[Gabriele Svelto [:gsvelto]]

If the bug is caused by an injected DLL we usually follow this procedure. That doesn't seem to be the case here though, is the AV actually blocking the file read? I don't think we can do much about it. That being said I suspect there's something else going wrong here since there's quite a few recent crash reports coming from Android and I doubt that it's an AV problem there.

Comment 3

[Marco Castelluccio [:marco]]

(In reply to Gabriele Svelto [:gsvelto] from comment #2)

If the bug is caused by an injected DLL we usually follow this procedure. That doesn't seem to be the case here though, is the AV actually blocking the file read? I don't think we can do much about it. That being said I suspect there's something else going wrong here since there's quite a few recent crash reports coming from Android and I doubt that it's an AV problem there.

I think I remember similar problems caused by some AV in the past. Maybe the Android crashes have the same signature but are not the same crash? After all, this crash can happen any time there is a corrupted installation, and corruptions can have different causes.
I guess we should try to contact Cisco to report this problem.

Comment 4

[Romain Testard [:RT]]

I'm asking people at Cisco to help find someone who can help there - sounds like we have evidence that AMP is the root cause per https://bugzilla.mozilla.org/show_bug.cgi?id=1681745#c0

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Camelia, please provide further information:

• Did this issue affect only one machine or multiple?
• What are the exact builds (version + build ID) which are affected?
• Can you share the omni.ja files it complains about? They are too big to get attached here, e.g. use Google Drive.

Comment 6

[Camelia Badau [:cbadau], Desktop Test Engineering]

Initially, the problem was reproduced on several machines with Cisco AMP, but today we retested and we have only one machine on which the problem still reproduces. Here, on this machine, the problem is reproduced only on the old ESR78.6.0 build installed on Dec 10, buildID: 20201207224150 (here is the omni.ja file for this build).
For new installs of ESR78.6.0 (or any other build: Release, Nightly) on this machine, everything works ok and the Firefox opens correctly.

On other machines, we deleted all Mozilla folders from AppData (Local, LocalLow and Roaming), and so the problem does not recur anymore.

Comment 7

[ivberlin]

Hi Camelia,
I'm a Cisco TSA covering AMP in Europe. I have not been able to reproduce your issue neither with the latest connector version (7.3.5) neither with the one you are currently using (7.2.11).
As you may know there are multiple prevention engines inside AMP. Would you accept i contact you directly so you could share with me more details about it? (the detailed event in the AMP console would be more than welcome).
Thank you
Ivan

Comment 8

[Camelia Badau [:cbadau], Desktop Test Engineering]

(In reply to ivberlin from comment #7)

Hi Camelia,
I'm a Cisco TSA covering AMP in Europe. I have not been able to reproduce your issue neither with the latest connector version (7.3.5) neither with the one you are currently using (7.2.11).
As you may know there are multiple prevention engines inside AMP. Would you accept i contact you directly so you could share with me more details about it? (the detailed event in the AMP console would be more than welcome).
Thank you
Ivan

Hi Ivan,

Sure, you can contact me directly on [e-mail].

Comment 9

[Wayne Mery (:wsmwk)]

Also startup crash for Thunderbird 78. bp-c744e5bf-1993-41b3-8325-a960b0210319

Comment 10

[[e-mail]]

I am running into this issue on a vmWare Horizon virtual machine on Win10 20H2.

What tends to happen is that I launch Firefox during login and it starts up just fine. Then if Firefox crashes or I accidentally close it during the day, it refuses to launch again and gives the below error. It won't even launch in Safe Mode (firefox.exe -safe-mode). The only option is to log-off from the virtual machine, wait for my session to be destroyed and re-login. Our vmWare administrators use something called AppSense to sync the necessary files. I also lack administrative rights on the virtual machine.

I don't see any security software (McAfee, Norton) or Cisco AMP but there is Seclore Endpoint.

crash.main.3
1628933812
c624822c - 59bb - 416b - 996d - 4e58395e491a{
"BuildID" : "20210706194121",
"CPUMicrocodeVersion" : "0x2000065",
"DOMIPCEnabled" : "1",
"InstallTime" : "1626327210",
"ProductID" : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
"ProductName" : "Firefox",
"ReleaseChannel" : "esr",
"SafeMode" : "0",
"ServerURL" : "https://crash-reports.mozilla.com/submit?id={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&version=78.12.0&buildid=20210706194121",
"StartupCrash" : "1",
"StartupTime" : "1628933811",
"Vendor" : "Mozilla",
"Version" : "78.12.0",
"CrashTime" : "1628933812",
"UptimeTS" : "16.24939",
"SecondsSinceLastCrash" : "102",
"BreakpadReserveAddress" : "2552009785344",
"BreakpadReserveSize" : "83886080",
"MozCrashReason" : "LoadSheetSync failed with error 80520012 loading built-in stylesheet 'chrome://global/content/xul.css'",
"ThreadIdNameMapping" : "10732:Gecko_IOThread,2116:Timer,14000:Link Monitor #1,11296:Socket Thread,13932:Permission,9320:JS Watchdog,13088:JS Helper,12124:JS Helper,10668:JS Helper,13480:BackgroundThreadPool #1,13676:IPDL Background,"
}

Comment 11

[Wayne Mery (:wsmwk)]

There is a recent sharp and significant increase - 4x overall - so now Firefox 102.4.0esr and 102.5.0esr this ranks #21. #49 for Thunderbird 102.5.0. 99% startup crashes.

Comment 12

[Haik Aftandilian [:haik]]

The majority of the recent crash reports I looked at had a Trend Micro module loaded, although not all of them.

@Greg, could you take a look at this one? This might be a case of many different third-party issues causing the same crash.

Comment 13

[Greg Stoll :gstoll]

Yeah, I'll take a look!

Comment 14

[Greg Stoll :gstoll]

The spike from a few months ago seems to be only in the ESR channel - we still see crashes in release, etc., but those appear steady over time.

Comment 15

[BugBot [:suhaib / :marco]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 20 desktop browser crashes on release (startup)

:gstoll, could you consider increasing the severity of this top-crash bug?

For more information, please visit BugBot documentation.

Comment 16

[Greg Stoll :gstoll]

The most recent spike in these crashes is in the release branch, and it corresponds with the release date of 113.0...but 75% of the crashes are from 112.0.2. I wonder if this indicates that people with 112.0.2 were trying to update to 113.0, the update failed somehow but left files in a bad state, and then launching the old version led to this crash?

Passing this on to Installers to see if this is plausible.

Comment 17

[Wayne Mery (:wsmwk)]

Crash rate dropped in 113. Probably no longer a topcrash.
https://crash-stats.mozilla.org/signature/?signature=mozilla::ErrorLoadingSheet&date=>=2023-05-28T00:42:00.000Z&date=<2023-06-04T00:42:00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

Comment 18

[BugBot [:suhaib / :marco]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit BugBot documentation.

Comment 19

[Gabriele Svelto [:gsvelto]]

There is a very large spike here of Windows 10 users with the following crash reason:

LoadSheetSync failed with error 80520012 loading built-in stylesheet 'resource://gre/res/designmode.css'

80520012 is NS_ERROR_FILE_NOT_FOUND and it's a startup crash, so the users are livid in the comments. The crash seems to be affecting only users on version 133.0.3. There's only a handful of crashes on file for version 134.0 and only one of them is for Windows 10 but the cause is different, so this might not be affecting people on 134.0. However I wonder if those experiencing the crash can upgrade or are stuck with the broken version.

Andrew what do you think? Is it worth to open an incident for this?

Comment 20

[Andrew McCreight [:mccr8]]

(In reply to Gabriele Svelto [:gsvelto] f

Andrew what do you think? Is it worth to open an incident for this?

I'm not sure, though it does look bad. I'd ask in #release-coordination on Slack. They could at least stop the update. I did see a few comments talking about how Firefox wouldn't start since they updated. I don't know how we'll make any progress on understanding the problem here.

Comment 21

[Andrew McCreight [:mccr8]]

I brought this up on Slack and there is some discussion there. Ryanvm mentioned bug 1388910, which deleted designmode.css. I found one crash that actually had a telemetry environment and it has the value 134.0 for displayVersion but 133.0.3 for version and platformVersion.

Comment 22

[Andrew McCreight [:mccr8]]

Looking at how the telemetry values are generated, I think displayVersion comes from the omnija (via AppConstants) and version and platformVersion come from the executable (via nsXULAppInfo), so that would be consistent with the mixed installation theory.

Comment 23

[Andrew McCreight [:mccr8]]

I split off a fresh bug for this crash spike to reduce confusion, bug 1941134.

Comment 24

[Pascal Chevrel (relman team) -> :pascalc]

This is back as a top startup crash with the release of Firefox 153.0:

(100.0% in signature vs 28.25% overall) moz_crash_reason = LoadSheetSync failed with error 80520012 loading built-in stylesheet 'resource://gre-resources/select.css'
(100.0% in signature vs 30.60% overall) dom_fission_enabled = 0

Comment 25

[Pascal Chevrel (relman team) -> :pascalc]

Duncan, can you help or find somebody who could help? If the volume keeps on spiking today, we may have to stop the rollout of 153.0. Thanks

Comment 26

[Pascal Chevrel (relman team) -> :pascalc]

I asked Claude to analyze this crash and here is its analysis, in case it helps:

Root cause (same class as bug 1941134): this is not a genuine 153 code regression — it's the recurring "updater leaves the executable and omni.ja on different versions" problem, surfacing through a different built-in
stylesheet each time (xul.css originally, designmode.css in the 133→134 spike / bug 1941134, now resource://gre-resources/select.css).

Mechanism: the set of built-in UA/Author stylesheets is compiled into native code (layout/style/BuiltInStyleSheetList.inc), while the .css files ship in omni.ja. The native entry and the jar.mn packaging line are always
added/removed in the same commit, so a self-consistent build never fails — a mismatch is only possible when the running binary and omni.ja come from different builds. When that happens, GlobalStyleSheetCache gets
NS_ERROR_FILE_NOT_FOUND (0x80520012) and ErrorLoadingSheet() intentionally MOZ_CRASHes (GlobalStyleSheetCache.cpp:527). It runs during the first presshell init, hence a startup crash.

Why select.css, why worse now: the Select built-in sheet churned across the 148–153 trains — added in bug 2011709, then reworked/reverted/re-landed in bug 2012698 (and touched by bug 2028130). That flip-flopping widens the
version window in which a binary and an omni.ja disagree about whether select.css exists, so the mismatch volume is higher than the designmode cycle.

To confirm the direction (as done in bug 1941134 comments 1 and 3): can we check version vs displayVersion/platformVersion on a sample of reports? A mismatch there is the smoking gun for the update-file-mismatch. Note the
dom_fission_enabled = 0 @ 100% correlation — I suspect this is a marker of the broken install/profile state rather than causal, but worth a glance.

Mitigation options:

1. Short-term "wallpaper" as in bug 1941134 — ensure select.css is present in the shipping omni.ja. Caveat: this only helps if the missing copy is in a build whose omni.ja we still control going forward; if the mismatch is
   new-binary + old-omni.ja, we can't retroactively patch the old package, so we need the direction (above) first.
2. Preferred real fix: select.css, details.css and accessiblecaret.css are Author-origin built-ins, not critical UA sheets. A missing Author built-in should not hard-crash startup — downgrade the failure from eCrash to
   eLogToConsole (with an empty-sheet fallback) for Author-origin sheets. That removes this whole class of startup crash regardless of which sheet churns next.
3. The underlying updater mismatch is the meta-bug (see bug 1917317 / 1909898 / 1014666).

Comment 27

[Duncan McIntosh [:dmcintosh]]

We've temporarily mitigated this, and I'd expect future developments to be in bug 1941972.

Comment 28

[BugBot [:suhaib / :marco]]

The bug is marked as blocking firefox153 (release). However, the bug still isn't assigned.

:Amir, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 29

[Daniel Holbert [:dholbert]]

Some users (at least one user) appear to trigger a secondary (and also-mysterious) startup-crash with a different signature (the one in bug 2056660) after triggering the crash here. See bug 2056660 comment 13 through bug 2056660 comment 16 for notes on that.

Comment 30

[Pascal Chevrel (relman team) -> :pascalc]

Resetting status flags changed by the landing of bug 2056926

Comment 31

[Nick Rishel [:nrishel]]

Clearing NI as this is being investigated.

Comment 33

[BugBot [:suhaib / :marco]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit BugBot documentation.

Comment 34

[AmirH [:Amir]]

Fixed as part of 153 incident