Open Bug 1337105 Opened 3 years ago Updated 2 years ago

Crash in OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured

Categories

(Firefox :: General, defect, P3, critical)

51 Branch
x86
Windows
defect

Tracking

()

Tracking Status
firefox-esr52 --- affected
firefox53 --- wontfix
firefox54 --- wontfix
firefox55 - wontfix
firefox56 + wontfix
firefox57 + wontfix
firefox58 + wontfix
firefox59 + unaffected
firefox60 + unaffected

People

(Reporter: adamam, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: crash, topcrash, Whiteboard: inj+)

Crash Data

This bug was filed from the Socorro interface and is 
report bp-002ffd52-9a94-46d1-a70d-4535f2170118.
=============================================================


Firefox crashes on launch. This is happening on our computers company-wide in the most recent FF update. Windows Event Log:

Faulting application name: firefox.exe, version: 51.0.1.6234, time stamp: 0x5888f28c
Faulting module name: KERNELBASE.dll, version: 6.1.7601.23572, time stamp: 0x57fd0379
Exception code: 0x406d1388
Fault offset: 0x0000c54f
Faulting process id: 0x241c
Faulting application start time: 0x01d280af49a2e5e8
Faulting application path: C:\Program Files (x86)\Mozilla Firefox\firefox.exe
Faulting module path: C:\WINDOWS\syswow64\KERNELBASE.dll
Report Id: 8770eb37-eca2-11e6-91e0-989096da2c83

I'm guessing it's going to be related to security software issues, as I have seen hints toward on other reports, so I will list what we use here: Microsoft System Endpoint Protection and Websense Filter (not Websense Endpoint Agent, which another Bugzilla report has referenced).
From a user in the company: installing 64-bit version of software got rid of the error. Seems to only persist in the 32-bit version.
Websense is probably the culprit here, and your case is surely another instance of bug 1313085.

Could you provide the version number of security apps installed on your machines, especially Websense.
Flags: needinfo?(adamam)
Keywords: crash
I'm having to play middleman between our Security Engineering department and you guys. I'll ask them and post the info here.
From one of our software engineers:

Vanilla Win7 image + Vanilla FF install = crash.
Changing the compatibility to Vista SP2 != crash
Windows 10 image + Vanilla FF install !=crash
Alright well they're not replying so here's the info I could scrounge up:

Microsoft System Center Endpoint Protection
Antimalware Client Version 4.10.205.0
Engine Version 1.1.13407.0
Netowkr Inspection System Engine Version 2.1.12706.0

Websense Filter: no instance installed on machines at OHSU. We use it to block people from reaching certain pages. I'm trying to get the security engineering dept to tell me more about this. From them when I asked for software details: "there is no Websense instance installed on any machines here at OHSU.  What they are referring to is like the TRITON AP-Endpoint software which is a fat client that is installed upon the machine to do things such as USB compliance and off-site (cloud) proxy capabilities.  Not something we have invested in."
Flags: needinfo?(adamam)
Yeah, this one is unrelated to WebSense.

Could it be bug 1335086? I see you have an old version of EMET.
No longer blocks: websense
See Also: → 1335086
Status: UNCONFIRMED → NEW
Ever confirmed: true
I'm not sure who manages that piece of software in our org so I've sent an email out to one of the people the manages OS and general system software deployment to see if he knows more about it and if we can get it updated for troubleshooting.
Well, as usual, the rest of the IT department at my org is refusing to cooperate and just update EMET, and instead is pushing the ESR version. So I won't be able to troubleshoot this further with you guys, but it most likely is EMET.
This is currently top crash #1.

#1
18.8% 	-18.62% 	
OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured

Count   Win    Mac Lin Ins isGC
1111 	1111 	0 	0 	2 	0 	2016-05-17 	1337105 1257387

#2
2.67% 	-0.57% 	
EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER

Count   Win Mac Lin Ins isGC
158 	0 	0 	0 	67 	3 	2013-11-14 	1360392 1279269 1255050 1245570 945328 837835 711568 610551
Signature report for:
OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured

Showing results from 7 days ago

Operating System
Windows 7 	3100 	35.0%
Windows Vista 	2067 	23.3%
Windows 10 	1971 	22.3%
Windows 8.1 	1436 	16.2%
Windows XP 	176 	2.0%
Windows 8 	74 	0.8%
OS X 10.11 	17 	0.2%
Linux 	        8 	0.1%
OS X 10.9 	2 	0.0%
OS X 10.12 	1 	0.0%
WindowsServ2003 1 	0.0%

Product
Firefox 	53.0 	2372 	26.9% 	2111
Firefox 52.1.0esr 	1689 	19.1% 	1188
Firefox 	55.0a1 	1118 	12.7% 	6
Firefox 52.1.1esr 	904 	10.2% 	909
Firefox 	53.0.2 	868 	9.8% 	834
Firefox 	54.0b4 	388 	4.4% 	244
Firefox 	52.0.2 	213 	2.4% 	226
Firefox 	54.0b3 	153 	1.7% 	116
Firefox 	54.0b5 	129 	1.5% 	92
Firefox 52.0.2esr 	76 	0.9% 	84
Firefox 	54.0a2 	37 	0.4% 	35
Firefox 	54.0b2 	37 	0.4% 	37
Firefox 52.0.1esr 	36 	0.4% 	52
Thunderbird 	52.1.0 	338 	3.8% 	312
Thunderbird 	52.0.1 	136 	1.5% 	108
Thunderbird 	52.0 	11 	0.1% 	10
Thunderbird 	53.0b2 	1 	0.0% 	1
SeaMonkey 	2.46 	25 	0.3% 	16
SeaMonkey 	2.50a2 	1 	0.0% 	1

Process Type
content 	2267 	25.6%

Uptime Range
> 1 hour 	3841 	43.4%
< 1 min 	1976 	22.3%
15-60 min 	1489 	16.8%
1-5 min 	823 	9.3%
5-15 min 	724 	8.2%

Architecture
x86 	7682 	86.8%
amd64 	1171 	13.2%

Flash Version
[blank] 	8853 	100.0%
(In reply to AmandaMarie Adams from comment #1)
> From a user in the company: installing 64-bit version of software got rid of
> the error. Seems to only persist in the 32-bit version.
Keywords: topcrash
OS: Windows 7 → Windows
Hardware: x86_64 → x86
(In reply to AmandaMarie Adams from comment #8)
> Well, as usual, the rest of the IT department at my org is refusing to
> cooperate and just update EMET, and instead is pushing the ESR version. So I
> won't be able to troubleshoot this further with you guys, but it most likely
> is EMET.
Summary: Crash in OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured → Crash in OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured [related to Microsoft's Enhanced Mitigation Experience Toolkit (EMET)]
See Also: → 1365207
Duplicate of this bug: 1365207
Doesn't look like this warrants tracking for 55.
[Tracking Requested - why for this re

This bug is a Top-Crasher.
Track 57+ as top crash. The volume of crashes in 56 seems low. Track 56- for now.
There are 5000+ crashes in the last week on 56 release for half as many installations. The pattern looks similar for 57.

Looks like there is a strong correlation with Symantec Endpoint. I'll try emailing our contacts there. 
Tracking this for 56/57, as maybe we can do something to blocklist older versions or prompt Symantec for a fix.
Jimm, do you agree this now looks like the crash is correlated with Symantec rather than with EMET? Should this be a Firefox::General bug or is there a better component?
Flags: needinfo?(jmathies)
Looking back at the crash-stats correlations I'm not sure.... Norton is also installed for many of the users who are crashing.
Yes heavily correlated now with the symantec dll IPSEng32.dll. We should reach out to them if we haven't already. ni'ing Adam on that.
Blocks: injecteject
Flags: needinfo?(jmathies) → needinfo?(astevenson)
Summary: Crash in OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured [related to Microsoft's Enhanced Mitigation Experience Toolkit (EMET)] → Crash in OOM | large | js::AutoEnterOOMUnsafeRegion::crash | js::AutoEnterOOMUnsafeRegion::crash | js::TenuringTracer::moveToTenured
Flags: needinfo?(astevenson)
Priority: -- → P2
I've reached out to Symantec on this.
I see they are investigating now.
Jim, any followup here from Symantec? This is still a high volume crash on release, ESR, and beta, though it seems better on 56.0.2 than on 56.0.
Flags: needinfo?(jmathies)
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #23)
> Jim, any followup here from Symantec? This is still a high volume crash on
> release, ESR, and beta, though it seems better on 56.0.2 than on 56.0.

They haven't responded. I'll ping them again.
Flags: needinfo?(jmathies)
(In reply to Jim Mathies [:jimm] from comment #24)
> (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #23)
> > Jim, any followup here from Symantec? This is still a high volume crash on
> > release, ESR, and beta, though it seems better on 56.0.2 than on 56.0.
> 
> They haven't responded. I'll ping them again.

Our correlation here isn't as solid as it was. Now down to 24%:

(24.25% in signature vs 02.31% overall) Module "IPSEng32.dll" = true [31.68% vs 02.83% if startup_crash = 0]
Still tracking as the volume is still very high.
Blocks: 1435797
No longer blocks: injecteject
Priority: P2 → P3
Whiteboard: inj+
I don't see any crashes with this signature for 59 and 60, in the last two weeks. 
Volume is high on 52.6.0esr, though.
This is still showing up really high on the ESR52 topcrash list. Symantec still looks like a pretty high correlation (Module "IPSEng32.dll" = true [57.43% vs 19.22% if platform_pretty_version = Windows Vista]). Jim, is there any chance we could try poking them again?
Flags: needinfo?(jmathies)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #28)
> This is still showing up really high on the ESR52 topcrash list. Symantec
> still looks like a pretty high correlation (Module "IPSEng32.dll" = true
> [57.43% vs 19.22% if platform_pretty_version = Windows Vista]). Jim, is
> there any chance we could try poking them again?

They responded to this asking for dumps to debug which we didn't have at the time. Might be able to get one now, will do some crash stats searching.
Flags: needinfo?(jmathies)
You need to log in before you can comment on or make changes to this bug.