812186 - crash in js::InvokeKernel @ UserCallWinProcCheckWow with Norton Confidential 2013

Status: RESOLVED WONTFIX

(Firefox :: Extension Compatibility, defect)

Description

[Scoobidiver (away)]

For Windows 8 only, it's #8 top crasher in 16.0.2. See https://crash-analysis.mozilla.com/rkaiser/2012-11-14/2012-11-14.firefox.16.0.2.win8.topcrash.html

The Windows 8 part is correlated to Norton Confidential 2013:
     42% (118/280) vs.   1% (2486/166702) coFFPlgn.dll
          0% (1/280) vs.   0% (318/166702) 2011.7.13.2
          0% (1/280) vs.   1% (1579/166702) 2012.5.7.2
          0% (0/280) vs.   0% (12/166702) 2013.1.3.1
         41% (116/280) vs.   0% (577/166702) 2013.2.0.18

Signature 	UserCallWinProcCheckWow More Reports Search
UUID	bb952dd9-7d8a-42d5-9b4c-d06e42121114
Date Processed	2012-11-14 21:18:34
Uptime	12833
Last Crash	5.0 days before submission
Install Age	2.7 weeks since version was first installed.
Install Time	2012-10-26 23:35:30
Product	Firefox
Version	16.0.2
Build ID	20121024073032
Release Channel	release
OS	Windows NT
OS Version	6.2.9200
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 37 stepping 2
Crash Reason	EXCEPTION_ACCESS_VIOLATION_EXEC
Crash Address	0x6668b179
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x0046, AdapterSubsysID: 1425103c, AdapterDriverVersion: 8.15.10.2858
D2D? D2D+ DWrite? DWrite+ D3D10 Layers? D3D10 Layers+ 
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x0046
Total Virtual Memory	4294836224
Available Virtual Memory	3440345088
System Memory Use Percentage	47
Available Page File	2476638208
Available Physical Memory	2140880896

Frame 	Module 	Signature 	Source
0 		@0x6668b179 	
1 	user32.dll 	UserCallWinProcCheckWow 	
2 	user32.dll 	CallWindowProcAorW 	
3 	user32.dll 	CallWindowProcW 	
4 	symhtml.dll 	symhtml.dll@0x24b88 	
5 	symhtml.dll 	symhtml.dll@0x23143 	
6 	user32.dll 	InternalCallWinProc 	
7 	user32.dll 	GetRealWindowOwner 	
8 	user32.dll 	DispatchClientMessage 	
9 	user32.dll 	__fnINLPCREATESTRUCT 	
10 	ntdll.dll 	KiUserCallbackDispatcher 	
11 	ntdll.dll 	KiUserApcDispatcher 	
12 	user32.dll 	CreateWindowInternal 	
13 	user32.dll 	CreateWindowExW 	
14 	symhtml.dll 	symhtml.dll@0xf22e 	
15 	symhtml.dll 	symhtml.dll@0x23a9a 	
16 	symhtml.dll 	symhtml.dll@0x2046a 	
17 	couictlr.dll 	couictlr.dll@0x24102 	
18 	couictlr.dll 	couictlr.dll@0x2441c 	
19 	cowpplg.dll 	cowpplg.dll@0x1689a 	
20 	cowpplg.dll 	cowpplg.dll@0x6881b 	
21 	cowpplg.dll 	cowpplg.dll@0x73638 	
22 	coFFPlgn.dll 	coFFPlgn.dll@0x355b9 	
23 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:344
24 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2442
25 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:355
26 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:387
27 	mozjs.dll 	JS_CallFunctionValue 	js/src/jsapi.cpp:5604
28 	xul.dll 	nsJSContext::CallEventHandler 	dom/base/nsJSEnvironment.cpp:1892
29 	xul.dll 	nsJSEventListener::HandleEvent 	dom/src/events/nsJSEventListener.cpp:188
30 	xul.dll 	nsEventListenerManager::HandleEventInternal 	content/events/src/nsEventListenerManager.cpp:895
31 	xul.dll 	nsEventTargetChainItem::HandleEventTargetChain 	content/events/src/nsEventDispatcher.cpp:316
32 	xul.dll 	nsEventDispatcher::Dispatch 	content/events/src/nsEventDispatcher.cpp:639
33 	xul.dll 	nsEventStateManager::DispatchMouseEvent 	content/events/src/nsEventStateManager.cpp:3835
34 	xul.dll 	nsEventStateManager::NotifyMouseOver 	content/events/src/nsEventStateManager.cpp:4010
35 	xul.dll 	nsEventStateManager::GenerateMouseEnterExit 	content/events/src/nsEventStateManager.cpp:4090
36 	xul.dll 	nsEventStateManager::PreHandleEvent 	content/events/src/nsEventStateManager.cpp:1145
37 	xul.dll 	PresShell::HandleEventInternal 	layout/base/nsPresShell.cpp:6436
38 	xul.dll 	PresShell::HandlePositionedEvent 	layout/base/nsPresShell.cpp:6187
39 	xul.dll 	PresShell::HandleEvent 	layout/base/nsPresShell.cpp:5984
...

More reports at:
https://crash-stats.mozilla.com/report/list?signature=UserCallWinProcCheckWow

Comment 1

[u279076]

Why is Firefox 16 affected but Firefox 17 wontfix?

Comment 2

[u279076]

I installed Firefox 16.0.2 and the free trial of Norton Internet Security 2013. I noticed immediately on startup that Norton Toolbar 2013.1.0.32 and Norton Vulnerability Protection 11.1.0.73-1 are disabled in Add-ons Manager because they are "incompatible with Firefox 16.0.2".

Is there anyway we can check to make sure users reporting these crashes do not have these add-ons enabled? Without being able to enabled these add-ons I can't find a way to get Norton to have any affect on my browsing experience whatsoever. I tried using the Identify Safe password manager as called out in some of the comments and it wouldn't work. Firefox's password manager was always used and Norton never kicked in at all.

I also tried some commonly used websites like Gmail, Yahoo, Youtube, and Facebook but I never experienced a crash.

Can someone please provide more details to better guide QA's investigation?

Comment 3

[u279076]

It's probably worth noting that Norton Confidential does not appear anywhere in Firefox even though Norton says it's a feature of their Internet Security suite.

Comment 4

[u279076]

Sorry, didn't mean to flip off the needinfo flag.

Comment 5

[u279076]

I've also tried installing Norton 360; same results as comment 2 and 3.

Installing Norton Antivirus asked me if I wanted to allow the Norton Identify Safe toolbar to be installed, so I allowed it. This resulted in Norton Identity Safe Toolbar 2013.2.0.18 being installed in Firefox. This allowed me to do some testing with the Identify Safe password manager but no crashing yet.

I'll keep trying to dogfood but it would be useful to have someone provide details so we can test this in a more guided manner.

Comment 6

[Scoobidiver (away)]

As stated in comment 0 and the whiteboard, it happens only on Windows 8. Crashes on other Windows versions are unrelated to this bug.

Comment 7

[u279076]

(In reply to Scoobidiver from comment #6)
> As stated in comment 0 and the whiteboard, it happens only on Windows 8.
> Crashes on other Windows versions are unrelated to this bug.

My testing was performed on Windows 8, sorry from not indicating this earlier.

Comment 8

[Virgil Dicu [:virgil] [QA]]

Mozilla/5.0 (Windows NT 6.2; rv:16.0) Gecko/20100101 Firefox/16.0 .2
Build ID: 20121024073032

Tried to reproduce the crash in Windows 8 x86 installing Norton Internet Security 2013 and Norton Antivirus. But same as comment 2, comment 3 and comment 5, nothing happened.

I uninstalled Firefox 16.0.2, restarted system and installed it back. Norton Identity Safe Toolbar 2012.0.18 and Norton Vulnerability Protection 11.1.11.5-3 toolbars were in the list of add-ons in Add-ons Manager. I enabled them, restarted Firefox, but still no crash after visiting Facebook, Gmail, Yahoo, Youtube. 

Is there anything else I could help here?

Comment 9

[Robert Kaiser]

This is the #20 crash on 17.0, with those add-on correlations:

  UserCallWinProcCheckWow|EXCEPTION_ACCESS_VIOLATION_EXEC (62 crashes)
     32% (20/62) vs.   1% (296/20686) {2D3F3651-74B9-4795-BDEC-6DA2F431CB62}
          0% (0/62) vs.   0% (30/20686) 2012.5.8.3
          0% (0/62) vs.   1% (161/20686) 2012.5.8.4
          2% (1/62) vs.   0% (3/20686) 2013.2.2.2
         31% (19/62) vs.   0% (102/20686) 2013.2.2.3
     27% (17/62) vs.   2% (350/20686) {BBDA0591-3099-440a-AA10-41764D9DB4DB}
         27% (17/62) vs.   2% (347/20686) 11.1.1.5 - 3
          0% (0/62) vs.   0% (3/20686) 2.0

Both those apparently are Norton stuff.

CCing Marcia as well, as she has some experience with testing around Norton/Symantec stuff.

Comment 10

[Robert Kaiser]

Still #21 on 17.0.1, only #173 on 18.0b2 though.

Comment 11

[Scoobidiver (away)]

(In reply to Robert Kaiser (:[e-mail]) from comment #10)
> only #173 on 18.0b2 though.
It's not correlated to Norton Confidential in 18.0 Beta:
  UserCallWinProcCheckWow|EXCEPTION_ACCESS_VIOLATION_EXEC (38 crashes)
     76% (29/38) vs.   8% (2903/35645) ksuser.dll (MS Windows DLL)
     63% (24/38) vs.   0% (53/35645) mono-1-vc.dll (Unity Web Player)
     63% (24/38) vs.   0% (74/35645) npUnity3D32.dll (Unity Web Player)

Comment 12

[Scoobidiver (away)]

It's #8 top crasher in 18.0.1 for Windows 8 only: https://crash-analysis.mozilla.com/rkaiser/2013-02-03/2013-02-03.firefox.18.0.1.win8.topcrash.html

It's still correlated to the latest version of Norton 2013:
     61% (202/329) vs.   2% (4149/174276) coFFPlgn.dll
          0% (1/329) vs.   2% (3441/174276) 2012.5.10.1
          0% (1/329) vs.   0% (4/174276) 2013.2.1.33
          8% (27/329) vs.   0% (135/174276) 2013.2.1.36
          4% (14/329) vs.   0% (37/174276) 2013.2.3.1
         48% (159/329) vs.   0% (532/174276) 2013.2.4.2

Comment 13

[Scoobidiver (away)]

It occurs also on Windows 7.

Comment 14

[Robert Kaiser]

This signatures has been rising on 19.0.2 in the last days, and it looks like that's this bug to a large degree, according to those correlations to Symantec libraries:

47% (382/809) vs.   1% (1212/172259) wincfi39.dll
45% (364/809) vs.   1% (1650/172259) ccl120u.dll
44% (353/809) vs.   0% (684/172259) symhtml.dll

Comment 15

[u279076]

Looks like this bug feel off our radar. Scoobidiver can you check to see what the current state of this crash is in Socorro?

Comment 16

[Scoobidiver (away)]

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #15)
> Looks like this bug feel off our radar. Scoobidiver can you check to see
> what the current state of this crash is in Socorro?
There are 2172 crashes in 22.0 whose 76% are correlated with Norton 2013:
     76% (267/350) vs.   1% (908/104870) wincfi39.dll (3.9.1.0)
     76% (267/350) vs.   1% (1167/104870) coFFPl22.dll
          0% (0/350) vs.   0% (244/104870) 2012.5.15.2
         76% (267/350) vs.   1% (922/104870) 2013.4.1.2
          0% (0/350) vs.   0% (1/104870) 2014.0.0.21
i.e. 1651 crashes and that makes it #31 browser crasher.

Comment 17

[Ioana (away)]

Removing keyword since QA already answered here...

This crash is still reproducible on current versions:
https://crash-stats.mozilla.com/report/list?signature=UserCallWinProcCheckWow&product=Firefox&query_type=contains&range_unit=weeks&process_type=any&hang_type=any&date=2014-01-15+14%3A00%3A00&range_value=4#reports

Comment 18

[u279076]

This crash has risen 5 spots to #20 in Firefox 28. This has gone up and down for several releases now. What is needed to move this forward?

Comment 19

[Andrei Vaida [:avaida]]

I tried to reproduce the crash on Windows 7 32-bit with:
- Firefox 26.0 (Build ID: 20131205075310) [1], 
- Nightly 29.0a1 (Build ID: 20140109030203) [2],
- the latest Beta (Build ID: 20140227174348) [3], 
with no success. 

Among other scenarios relevant to the data available in socorro for this issue, I also targeted websites that heavily use Unity's web player (e.g. Facebook games, bikeconfig.com), still, with no results.

[1] Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0
[2] Mozilla/5.0 (Windows NT 6.1; rv:29.0) Gecko/20100101 Firefox/29.0
[3] Mozilla/5.0 (Windows NT 6.1; rv:28.0) Gecko/20100101 Firefox/28.0

Comment 20

[Lukas Blakk [:lsblakk] use ?needinfo]

Longstanding stability issue - would love to see some movement on here, but it's not a blocker for shipping FF28.

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Could we pull together any open stability bugs that might be related to Norton and then we can send them a roll-up inquiry about the batch?  I'm assuming we have more than just this one, so might as well make the outreach more effective.

Comment 22

[u279076]

(In reply to Lukas Blakk [:lsblakk] from comment #21)
> Could we pull together any open stability bugs that might be related to
> Norton and then we can send them a roll-up inquiry about the batch?

I'll have to rely on KaiRo for this request as I'm not sure how to do this.

Comment 23

[Robert Kaiser]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #22)
> (In reply to Lukas Blakk [:lsblakk] from comment #21)
> > Could we pull together any open stability bugs that might be related to
> > Norton and then we can send them a roll-up inquiry about the batch?
> 
> I'll have to rely on KaiRo for this request as I'm not sure how to do this.

Me neither, really. That's why it so long for me to reply. (Also, the whole request shouldn't go in this bug itself, IMHO, but on email.)
I can't think of anything other than https://bugzilla.mozilla.org/buglist.cgi?keywords=crash&keywords_type=allwords&short_desc=norton&short_desc_type=allwordssubstr&resolution=--- to figure out this one, and I have no idea what of those are stale or not and going through them with https://crash-stats.mozilla.com/topcrasher_ranks_bybug/ takes ages because of the inefficient implementation we have at this time.
Some Norton problems tend to flare up shortly after release (though nowadays Symantec should be testing on Beta and get their ducks in a row before we ship the release).
The rather generic signature of this specific bug is not even correlated to Norton on 28 beta (it is on 27 release) but instead to something called K7OEPlgn.dll.

Comment 24

[u279076]

Marking wontfix for Firefox 28 and earlier given we are releasing tomorrow.

Comment 25

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

Norton Confidential 2013 is obsolete.