1337105 - Crash in OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured

Status: RESOLVED WORKSFORME

(Firefox :: General, defect, P3)

Description

[AmandaMarie Adams]

This bug was filed from the Socorro interface and is 
report bp-002ffd52-9a94-46d1-a70d-4535f2170118.
=============================================================


Firefox crashes on launch. This is happening on our computers company-wide in the most recent FF update. Windows Event Log:

Faulting application name: firefox.exe, version: 51.0.1.6234, time stamp: 0x5888f28c
Faulting module name: KERNELBASE.dll, version: 6.1.7601.23572, time stamp: 0x57fd0379
Exception code: 0x406d1388
Fault offset: 0x0000c54f
Faulting process id: 0x241c
Faulting application start time: 0x01d280af49a2e5e8
Faulting application path: C:\Program Files (x86)\Mozilla Firefox\firefox.exe
Faulting module path: C:\WINDOWS\syswow64\KERNELBASE.dll
Report Id: 8770eb37-eca2-11e6-91e0-989096da2c83

I'm guessing it's going to be related to security software issues, as I have seen hints toward on other reports, so I will list what we use here: Microsoft System Endpoint Protection and Websense Filter (not Websense Endpoint Agent, which another Bugzilla report has referenced).

Comment 1

[AmandaMarie Adams]

From a user in the company: installing 64-bit version of software got rid of the error. Seems to only persist in the 32-bit version.

Comment 2

[Loic]

Websense is probably the culprit here, and your case is surely another instance of bug 1313085.

Could you provide the version number of security apps installed on your machines, especially Websense.

Comment 3

[AmandaMarie Adams]

I'm having to play middleman between our Security Engineering department and you guys. I'll ask them and post the info here.

Comment 4

[AmandaMarie Adams]

From one of our software engineers:

Vanilla Win7 image + Vanilla FF install = crash.
Changing the compatibility to Vista SP2 != crash
Windows 10 image + Vanilla FF install !=crash

Comment 5

[AmandaMarie Adams]

Alright well they're not replying so here's the info I could scrounge up:

Microsoft System Center Endpoint Protection
Antimalware Client Version 4.10.205.0
Engine Version 1.1.13407.0
Netowkr Inspection System Engine Version 2.1.12706.0

Websense Filter: no instance installed on machines at OHSU. We use it to block people from reaching certain pages. I'm trying to get the security engineering dept to tell me more about this. From them when I asked for software details: "there is no Websense instance installed on any machines here at OHSU.  What they are referring to is like the TRITON AP-Endpoint software which is a fat client that is installed upon the machine to do things such as USB compliance and off-site (cloud) proxy capabilities.  Not something we have invested in."

Comment 6

[Marco Castelluccio [:marco]]

Yeah, this one is unrelated to WebSense.

Could it be bug 1335086? I see you have an old version of EMET.

Comment 7

[AmandaMarie Adams]

I'm not sure who manages that piece of software in our org so I've sent an email out to one of the people the manages OS and general system software deployment to see if he knows more about it and if we can get it updated for troubleshooting.

Comment 8

[AmandaMarie Adams]

Well, as usual, the rest of the IT department at my org is refusing to cooperate and just update EMET, and instead is pushing the ESR version. So I won't be able to troubleshoot this further with you guys, but it most likely is EMET.

Comment 9

[Trevor Skywalker]

This is currently top crash #1.

#1
18.8% 	-18.62% 	
OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured

Count   Win    Mac Lin Ins isGC
1111 	1111 	0 	0 	2 	0 	2016-05-17 	1337105 1257387

#2
2.67% 	-0.57% 	
EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER

Count   Win Mac Lin Ins isGC
158 	0 	0 	0 	67 	3 	2013-11-14 	1360392 1279269 1255050 1245570 945328 837835 711568 610551

Comment 10

[Trevor Skywalker]

Signature report for:
OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured

Showing results from 7 days ago

Operating System
Windows 7 	3100 	35.0%
Windows Vista 	2067 	23.3%
Windows 10 	1971 	22.3%
Windows 8.1 	1436 	16.2%
Windows XP 	176 	2.0%
Windows 8 	74 	0.8%
OS X 10.11 	17 	0.2%
Linux 	        8 	0.1%
OS X 10.9 	2 	0.0%
OS X 10.12 	1 	0.0%
WindowsServ2003 1 	0.0%

Product
Firefox 	53.0 	2372 	26.9% 	2111
Firefox 52.1.0esr 	1689 	19.1% 	1188
Firefox 	55.0a1 	1118 	12.7% 	6
Firefox 52.1.1esr 	904 	10.2% 	909
Firefox 	53.0.2 	868 	9.8% 	834
Firefox 	54.0b4 	388 	4.4% 	244
Firefox 	52.0.2 	213 	2.4% 	226
Firefox 	54.0b3 	153 	1.7% 	116
Firefox 	54.0b5 	129 	1.5% 	92
Firefox 52.0.2esr 	76 	0.9% 	84
Firefox 	54.0a2 	37 	0.4% 	35
Firefox 	54.0b2 	37 	0.4% 	37
Firefox 52.0.1esr 	36 	0.4% 	52
Thunderbird 	52.1.0 	338 	3.8% 	312
Thunderbird 	52.0.1 	136 	1.5% 	108
Thunderbird 	52.0 	11 	0.1% 	10
Thunderbird 	53.0b2 	1 	0.0% 	1
SeaMonkey 	2.46 	25 	0.3% 	16
SeaMonkey 	2.50a2 	1 	0.0% 	1

Process Type
content 	2267 	25.6%

Uptime Range
> 1 hour 	3841 	43.4%
< 1 min 	1976 	22.3%
15-60 min 	1489 	16.8%
1-5 min 	823 	9.3%
5-15 min 	724 	8.2%

Architecture
x86 	7682 	86.8%
amd64 	1171 	13.2%

Flash Version
[blank] 	8853 	100.0%

Comment 11

[Trevor Skywalker]

(In reply to AmandaMarie Adams from comment #1)
> From a user in the company: installing 64-bit version of software got rid of
> the error. Seems to only persist in the 32-bit version.

Comment 12

[Trevor Skywalker]

(In reply to AmandaMarie Adams from comment #8)
> Well, as usual, the rest of the IT department at my org is refusing to
> cooperate and just update EMET, and instead is pushing the ESR version. So I
> won't be able to troubleshoot this further with you guys, but it most likely
> is EMET.

Comment 14

[Julien Cristau [:jcristau]]

Doesn't look like this warrants tracking for 55.

Comment 15

[Trevor Skywalker]

[Tracking Requested - why for this re

This bug is a Top-Crasher.

Comment 16

[Gerry Chang [:gchang]]

Track 57+ as top crash. The volume of crashes in 56 seems low. Track 56- for now.

Comment 17

[Liz Henry (:lizzard) (relman/hg->git project)]

There are 5000+ crashes in the last week on 56 release for half as many installations. The pattern looks similar for 57.

Looks like there is a strong correlation with Symantec Endpoint. I'll try emailing our contacts there. 
Tracking this for 56/57, as maybe we can do something to blocklist older versions or prompt Symantec for a fix.

Comment 18

[Liz Henry (:lizzard) (relman/hg->git project)]

Jimm, do you agree this now looks like the crash is correlated with Symantec rather than with EMET? Should this be a Firefox::General bug or is there a better component?

Comment 19

[Liz Henry (:lizzard) (relman/hg->git project)]

Looking back at the crash-stats correlations I'm not sure.... Norton is also installed for many of the users who are crashing.

Comment 20

[Jim Mathies [:jimm]]

Yes heavily correlated now with the symantec dll IPSEng32.dll. We should reach out to them if we haven't already. ni'ing Adam on that.

Comment 21

[Jim Mathies [:jimm]]

I've reached out to Symantec on this.

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

I see they are investigating now.

Comment 23

[Liz Henry (:lizzard) (relman/hg->git project)]

Jim, any followup here from Symantec? This is still a high volume crash on release, ESR, and beta, though it seems better on 56.0.2 than on 56.0.

Comment 24

[Jim Mathies [:jimm]]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #23)
> Jim, any followup here from Symantec? This is still a high volume crash on
> release, ESR, and beta, though it seems better on 56.0.2 than on 56.0.

They haven't responded. I'll ping them again.

Comment 25

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #24)
> (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #23)
> > Jim, any followup here from Symantec? This is still a high volume crash on
> > release, ESR, and beta, though it seems better on 56.0.2 than on 56.0.
> 
> They haven't responded. I'll ping them again.

Our correlation here isn't as solid as it was. Now down to 24%:

(24.25% in signature vs 02.31% overall) Module "IPSEng32.dll" = true [31.68% vs 02.83% if startup_crash = 0]

Comment 26

[Sylvestre Ledru [:Sylvestre]]

Still tracking as the volume is still very high.

Comment 27

[Liz Henry (:lizzard) (relman/hg->git project)]

I don't see any crashes with this signature for 59 and 60, in the last two weeks. 
Volume is high on 52.6.0esr, though.

Comment 28

[Ryan VanderMeulen [:RyanVM]]

This is still showing up really high on the ESR52 topcrash list. Symantec still looks like a pretty high correlation (Module "IPSEng32.dll" = true [57.43% vs 19.22% if platform_pretty_version = Windows Vista]). Jim, is there any chance we could try poking them again?

Comment 29

[Jim Mathies [:jimm]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #28)
> This is still showing up really high on the ESR52 topcrash list. Symantec
> still looks like a pretty high correlation (Module "IPSEng32.dll" = true
> [57.43% vs 19.22% if platform_pretty_version = Windows Vista]). Jim, is
> there any chance we could try poking them again?

They responded to this asking for dumps to debug which we didn't have at the time. Might be able to get one now, will do some crash stats searching.

Comment 30

[Wayne Mery (:wsmwk)]

No crashes nwer than version 75
https://crash-stats.mozilla.org/signature/?signature=CallTraceHook%3CT%3E&date=%3E%3D2021-04-16T20%3A44%3A00.000Z&date=%3C2021-07-16T20%3A44%3A00.000Z