Last Comment Bug 783369 - Norton Confidential 2012/2013 crashes (couictlr.dll, coffplgn.dll)
: Norton Confidential 2012/2013 crashes (couictlr.dll, coffplgn.dll)
Status: RESOLVED FIXED
[Fixed in Norton 2012.5.11.8 and 2013...
: crash, topcrash
Product: Firefox
Classification: Client Software
Component: Extension Compatibility (show other bugs)
: 18 Branch
: x86 Windows 7
: -- critical with 4 votes (vote)
: ---
Assigned To: Jorge Villalobos [:jorgev]
: Anthony Hughes (:ashughes) [GFX][QA][Mentor]
Mentors:
: 829929 830107 831101 833216 835092 836763 (view as bug list)
Depends on: 828296 831122
Blocks: Norton
  Show dependency treegraph
 
Reported: 2012-08-16 13:41 PDT by Marcia Knous [:marcia - use ni]
Modified: 2013-02-11 09:03 PST (History)
18 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
wontfix
affected
-
-


Attachments

Description Marcia Knous [:marcia - use ni] 2012-08-16 13:41:10 PDT
This bug was filed from the Socorro interface and is 
report bp-42a00953-b4e3-4ab7-a492-9a0bc2120816 .
============================================================= 

We have seen a bug like this previously - Bug 492437. https://crash-stats.mozilla.com/report/list?signature=couictlr.dll%400x26373 to the crashes which are all Windows. We should reach out to Norton to try to get some traction on this issue. 

2012.5.5.11 seems to the problematic dll from the correlation reports:

    100% (525/525) vs.   1% (2193/183047) coFFPlgn.dll
          0% (0/525) vs.   0% (440/183047) 2011.7.10.1
         98% (517/525) vs.   1% (1247/183047) 2012.5.5.11
          0% (0/525) vs.   0% (8/183047) 2012.6.3.28
          2% (8/525) vs.   0% (495/183047) 2012.7.4.1
          0% (0/525) vs.   0% (1/183047) 2013.1.0.11
          0% (0/525) vs.   0% (2/183047) 2013.1.0.8
    100% (524/525) vs.   1% (2077/183047) cowpplg.dll
          0% (0/525) vs.   0% (434/183047) 2011.7.9.4
          0% (0/525) vs.   0% (368/183047) 2012.5.4.6
        100% (524/525) vs.   1% (1267/183047) 2012.5.5.11
          0% (0/525) vs.   0% (8/183047) 2012.6.3.28
    100% (524/525) vs.   1% (2077/183047) couictlr.dll
          0% (0/525) vs.   0% (434/183047) 2011.7.9.4
          0% (0/525) vs.   0% (368/183047) 2012.5.4.6
        100% (524/525) vs.   1% (1267/183047) 2012.5.5.11
          0% (0/525) vs.   0% (8/183047) 2012.6.3.28

Frame 	Module 	Signature 	Source
0 	couictlr.dll 	couictlr.dll@0x26373 	
1 	coFFPlgn.dll 	coFFPlgn.dll@0x4b90b 	
2 	coFFPlgn.dll 	coFFPlgn.dll@0x4b667 	
3 	coFFPlgn.dll 	coFFPlgn.dll@0x37bca 	
4 	coFFPlgn.dll 	coFFPlgn.dll@0x3af19 	
5 	coFFPlgn.dll 	coFFPlgn.dll@0x43fe4 	
6 	coFFPlgn.dll 	coFFPlgn.dll@0x450ea 	
7 	xul.dll 	nsDocLoader::DoFireOnStateChange 	uriloader/base/nsDocLoader.cpp:1383
8 	xul.dll 	nsDocLoader::FireOnStateChange 	uriloader/base/nsDocLoader.cpp:1320
9 	xul.dll 	nsLoadGroup::SetDefaultLoadRequest 	netwerk/base/src/nsLoadGroup.cpp:523
10 	xul.dll 	nsDocLoader::doStartDocumentLoad 	uriloader/base/nsDocLoader.cpp:885
11 	xul.dll 	nsDocLoader::OnStartRequest 	uriloader/base/nsDocLoader.cpp:582
12 	xul.dll 	nsLoadGroup::AddRequest 	netwerk/base/src/nsLoadGroup.cpp:612
13 	xul.dll 	nsHttpChannel::AsyncOpen 	netwerk/protocol/http/nsHttpChannel.cpp:3924
14 	xul.dll 	nsURILoader::OpenURI 	uriloader/base/nsURILoader.cpp:815
15 	xul.dll 	nsDocShell::DoChannelLoad 	docshell/base/nsDocShell.cpp:9182
16 	xul.dll 	nsDocShell::DoURILoad 	docshell/base/nsDocShell.cpp:9019
17 	xul.dll 	nsDocShell::InternalLoad 	docshell/base/nsDocShell.cpp:8710
18 	xul.dll 	nsDocShell::LoadURI 	docshell/base/nsDocShell.cpp:1497
19 	xul.dll 	nsRefPtr<nsIDOMEventListener>::~nsRefPtr<nsIDOMEventListener> 	obj-firefox/dist/include/nsAutoPtr.h:908
20 	xul.dll 	nsFrameLoader::ReallyStartLoading 	content/base/src/nsFrameLoader.cpp:432
21 	xul.dll 	nsDocument::MaybeInitializeFinalizeFrameLoaders 	content/base/src/nsDocument.cpp:5505
22 	xul.dll 	nsHTMLDocument::EndUpdate 	content/html/document/src/nsHTMLDocument.cpp:2284
23 	xul.dll 	mozAutoDocUpdate::~mozAutoDocUpdate 	content/base/src/mozAutoDocUpdate.h:67
24 	xul.dll 	nsGenericHTMLElement::SetInnerHTML 	content/html/content/src/nsGenericHTMLElement.cpp:806
25 	xul.dll 	nsIDOMHTMLElement_SetInnerHTML 	obj-firefox/js/xpconnect/src/dom_quickstubs.cpp:14209
26 	mozjs.dll 	js::Shape::set 	js/src/jsscopeinlines.h:314
27 	mozjs.dll 	js_SetPropertyHelper 	js/src/jsobj.cpp:5339
28 	mozjs.dll 	js::SetPropertyOperation 	js/src/jsinterpinlines.h:349
29 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2671
30 	mozjs.dll 	js::RunScript 	js/src/jsinterp.cpp:475
31 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:535
32 	mozjs.dll 	js::CallOrConstructBoundFunction 	js/src/jsfun.cpp:840
33 	mozjs.dll 	js::mjit::CallCompiler::generateNativeStub 	js/src/methodjit/MonoIC.cpp:812
34 	mozjs.dll 	js::mjit::ic::NativeCall 	js/src/methodjit/MonoIC.cpp:1045
35 	mozjs.dll 	js::mjit::EnterMethodJIT 	js/src/methodjit/MethodJIT.cpp:1052
36 	mozjs.dll 	js::mjit::JaegerShot 	js/src/methodjit/MethodJIT.cpp:1123
37 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2803
38 	mozjs.dll 	js::RunScript 	js/src/jsinterp.cpp:475
39 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:535
40 	mozjs.dll 	js::CallOrConstructBoundFunction 	js/src/jsfun.cpp:840
41 	mozjs.dll 	js::mjit::CallCompiler::generateNativeStub 	js/src/methodjit/MonoIC.cpp:812
42 	mozjs.dll 	js::mjit::ic::NativeCall 	js/src/methodjit/MonoIC.cpp:1045
43 	mozjs.dll 	js::mjit::EnterMethodJIT 	js/src/methodjit/MethodJIT.cpp:1052
44 	mozjs.dll 	js::mjit::JaegerShot 	js/src/methodjit/MethodJIT.cpp:1123
45 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2803
46 	mozjs.dll 	UncachedInlineCall 	js/src/methodjit/InvokeHelpers.cpp:376
47 	mozjs.dll 	js::mjit::stubs::UncachedCallHelper 	js/src/methodjit/InvokeHelpers.cpp:459
48 	mozjs.dll 	js::mjit::stubs::CompileFunction 	js/src/methodjit/InvokeHelpers.cpp:287
49 	mozjs.dll 	js::mjit::EnterMethodJIT 	js/src/methodjit/MethodJIT.cpp:1052
50 	mozjs.dll 	js::mjit::JaegerShot 	js/src/methodjit/MethodJIT.cpp:1123
51 	mozjs.dll 	js::RunScript 	js/src/jsinterp.cpp:472
52 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:535
53 	mozjs.dll 	js::CallOrConstructBoundFunction 	js/src/jsfun.cpp:840
54 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:519
55 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2757
56 	mozjs.dll 	js::RunScript 	js/src/jsinterp.cpp:467
57 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:535
58 	mozjs.dll 	js::CallOrConstructBoundFunction 	js/src/jsfun.cpp:840
59 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:519
60 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:567
61 	mozjs.dll 	JS_CallFunctionValue 	js/src/jsapi.cpp:5429
62 	xul.dll 	nsJSContext::CallEventHandler 	dom/base/nsJSEnvironment.cpp:1893
63 	xul.dll 	nsCycleCollectingAutoRefCnt::incr 	obj-firefox/dist/include/nsISupportsImpl.h:161
64 	user32.dll 	CalcWakeMask 	
65 	xul.dll 	PeekUIMessage 	widget/windows/nsAppShell.cpp:102
66 	xul.dll 	TimerThread::UpdateFilter 	xpcom/threads/TimerThread.cpp:241
67 	xul.dll 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:508
68 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:656
69 	xul.dll 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:114
70 	xul.dll 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:201
71 	xul.dll 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:175
72 	xul.dll 	nsBaseAppShell::Run 	widget/xpwidgets/nsBaseAppShell.cpp:189
73 	xul.dll 	nsAppShell::Run 	widget/windows/nsAppShell.cpp:267
74 	xul.dll 	nsAppStartup::Run 	toolkit/components/startup/nsAppStartup.cpp:295
75 	xul.dll 	XREMain::XRE_mainRun 	toolkit/xre/nsAppRunner.cpp:3780
76 	xul.dll 	XREMain::XRE_main 	toolkit/xre/nsAppRunner.cpp:3857
77 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3933
78 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:107
79 	firefox.exe 	__tmainCRTStartup 	crtexe.c:552
80 	kernel32.dll 	BaseThreadInitThunk 	
81 	ntdll.dll 	__RtlUserThreadStart 	
82 	ntdll.dll 	_RtlUserThreadStart
Comment 1 Robert Kaiser (not working on stability any more) 2012-08-17 07:43:28 PDT
I'm adding the other 2012.5.5.11-raleted crash here that started rising at the same time:

  coffplgn.dll@0x3afa6|EXCEPTION_ACCESS_VIOLATION_READ (387 crashes)
    100% (387/387) vs.   2% (3252/181810) coFFPlgn.dll
          0% (0/387) vs.   0% (427/181810) 2011.7.10.1
        100% (387/387) vs.   1% (2647/181810) 2012.5.5.11
          0% (0/387) vs.   0% (5/181810) 2012.6.3.28
          0% (0/387) vs.   0% (172/181810) 2012.7.4.1
          0% (0/387) vs.   0% (1/181810) 2013.1.0.11


couictlr.dll@0x26373 is now #22 on 14.0.1, coffplgn.dll@0x3afa6 is #47 in yesterday's data, with 111 and 39 crashes per million ADI.
Comment 2 Lukas Blakk [:lsblakk] use ?needinfo 2012-08-20 16:51:01 PDT
Passing this to Jorge - do we need to look into blocklisting here?
Comment 3 Jorge Villalobos [:jorgev] 2012-08-21 07:46:16 PDT
Maybe. I'm looking for ways to get in touch with them before we take this route, though.
Comment 4 Jens S 2012-08-24 10:12:54 PDT
Did you guys collect any dumps? If yes, where can I find them?
Comment 5 Marcia Knous [:marcia - use ni] 2012-08-24 10:56:58 PDT
It might be coincidental, but with the couictlr.dll@0x26373 signature a fair number of the comments mention the crash happened when they were doing something related to flash.
Comment 6 Robert Kaiser (not working on stability any more) 2012-08-24 11:00:49 PDT
(In reply to Jens S from comment #4)
> Did you guys collect any dumps? If yes, where can I find them?

We have minidumps, but as they can contain privacy-related information, they're not publicly visible. You might need to go through our partner contacts to get access to some.
Comment 7 Marcia Knous [:marcia - use ni] 2012-08-29 11:11:04 PDT
We just shipped Firefox 15, and I see a signature showing up in early 15 data correlated to Norton Confidential that shows coFFplgn Version 2012.7.5.2 as being crashy. https://crash-stats.mozilla.com/report/list?signature=coffplgn.dll@0x3ac1b links to those crashes.
Comment 8 Tyler Downer [:Tyler] 2012-08-30 14:10:19 PDT
According to SUMO's research, Norton Confidential is defunct and replaced by Norton Identify Safe. Is that correct?
Comment 9 Marcia Knous [:marcia - use ni] 2012-09-06 14:50:21 PDT
#30 top crash on FF 15 at the moment.
Comment 10 Marcia Knous [:marcia - use ni] 2012-09-06 14:51:52 PDT
According to what is in this post - http://community.norton.com/t5/Norton-Toolbar-Norton-Identity/NIS-2012-has-Norton-Confidential/m-p/748130/highlight/true#M1742, Norton Confidential seems to be tied to the toolbar and is part of the NIS 2012 suite.

(In reply to Tyler Downer [:Tyler] from comment #8)
> According to SUMO's research, Norton Confidential is defunct and replaced by
> Norton Identify Safe. Is that correct?
Comment 11 Marcia Knous [:marcia - use ni] 2012-09-11 15:40:01 PDT
#24 top crash in Firefox 15.0.1. Many of the comments mention the crash happens when interacting with Flash and here are some of the URLs:

286 	about:blank
93 	https://www.facebook.com/
86 	http://www.facebook.com/
41 	http://www.aol.com/
39 	http://www.t-online.de/
37 	http://blog.livedoor.jp/ryodan13/
21 	http://www.yahoo.com/
20 	http://l.yimg.com/rq/darla/2-4-2/html/ext-render-secure.html
20 	about:newtab
19 	http://mail.aol.com/36962-112/aol-6/en-us/Suite.aspx
16 	https://www.facebook.com/?ref=tn_tnmn
15 	https://twitter.com/
15 	http://my.yahoo.com/
15 	http://www.youtube.com/
14 	https://platform.twitter.com/widgets/hub.html
14 	http://www.msn.com/
14 	http://www.google.com/ig
12 	http://xfinity.comcast.net/

Unfortunately due to the nature of the crash we are not capturing the flash version being used.
Comment 12 Robert Kaiser (not working on stability any more) 2012-10-03 07:20:27 PDT
I didn't find out the exact version involved with this new crash, but it looks like https://crash-stats.mozilla.com/report/list?signature=couictlr.dll%400x2d280 is yet another one of those signatures.
Comment 13 Scoobidiver (away) 2012-10-03 08:01:13 PDT
With combined signatures, it's #16 top browser crasher in 15.0.1.

Here are today's correlations per module version in 15.0.1:
  couictlr.dll@0x26373|EXCEPTION_ACCESS_VIOLATION_READ (538 crashes)
    100% (538/538) vs.   2% (2785/172615) coFFPlgn.dll
          0% (0/538) vs.   0% (357/172615) 2011.7.12.1
          0% (0/538) vs.   0% (642/172615) 2012.5.6.10
          0% (0/538) vs.   0% (5/172615) 2012.6.4.2
        100% (538/538) vs.   1% (1538/172615) 2012.7.5.2
          0% (0/538) vs.   0% (22/172615) 2013.1.0.32
          0% (0/538) vs.   0% (215/172615) 2013.1.1.4
          0% (0/538) vs.   0% (5/172615) 2013.1.1.7
          0% (0/538) vs.   0% (1/172615) 2013.2.0.4

  couictlr.dll@0x2d280|EXCEPTION_ACCESS_VIOLATION_READ (244 crashes)
    100% (244/244) vs.   2% (2785/172615) coFFPlgn.dll
          0% (0/244) vs.   0% (357/172615) 2011.7.12.1
         98% (240/244) vs.   0% (642/172615) 2012.5.6.10
          0% (0/244) vs.   0% (5/172615) 2012.6.4.2
          2% (4/244) vs.   1% (1538/172615) 2012.7.5.2
          0% (0/244) vs.   0% (22/172615) 2013.1.0.32
          0% (0/244) vs.   0% (215/172615) 2013.1.1.4
          0% (0/244) vs.   0% (5/172615) 2013.1.1.7
          0% (0/244) vs.   0% (1/172615) 2013.2.0.4

  coffplgn.dll@0x3ac1b|EXCEPTION_ACCESS_VIOLATION_READ (231 crashes)
    100% (231/231) vs.   2% (2785/172615) coFFPlgn.dll
          0% (0/231) vs.   0% (357/172615) 2011.7.12.1
          0% (0/231) vs.   0% (642/172615) 2012.5.6.10
          0% (0/231) vs.   0% (5/172615) 2012.6.4.2
        100% (231/231) vs.   1% (1538/172615) 2012.7.5.2
          0% (0/231) vs.   0% (22/172615) 2013.1.0.32
          0% (0/231) vs.   0% (215/172615) 2013.1.1.4
          0% (0/231) vs.   0% (5/172615) 2013.1.1.7
          0% (0/231) vs.   0% (1/172615) 2013.2.0.4

  coffplgn.dll@0x3ac40|EXCEPTION_ACCESS_VIOLATION_READ (65 crashes)
    100% (65/65) vs.   2% (2785/172615) coFFPlgn.dll
          0% (0/65) vs.   0% (357/172615) 2011.7.12.1
        100% (65/65) vs.   0% (642/172615) 2012.5.6.10
          0% (0/65) vs.   0% (5/172615) 2012.6.4.2
          0% (0/65) vs.   1% (1538/172615) 2012.7.5.2
          0% (0/65) vs.   0% (22/172615) 2013.1.0.32
          0% (0/65) vs.   0% (215/172615) 2013.1.1.4
          0% (0/65) vs.   0% (5/172615) 2013.1.1.7
          0% (0/65) vs.   0% (1/172615) 2013.2.0.4

  coffplgn.dll@0x3ac1f|EXCEPTION_ACCESS_VIOLATION_READ (20 crashes)
    100% (20/20) vs.   2% (2785/172615) coFFPlgn.dll
          0% (0/20) vs.   0% (357/172615) 2011.7.12.1
          0% (0/20) vs.   0% (642/172615) 2012.5.6.10
          0% (0/20) vs.   0% (5/172615) 2012.6.4.2
        100% (20/20) vs.   1% (1538/172615) 2012.7.5.2
          0% (0/20) vs.   0% (22/172615) 2013.1.0.32
          0% (0/20) vs.   0% (215/172615) 2013.1.1.4
          0% (0/20) vs.   0% (5/172615) 2013.1.1.7
          0% (0/20) vs.   0% (1/172615) 2013.2.0.4
Comment 14 Lukas Blakk [:lsblakk] use ?needinfo 2012-10-03 10:07:16 PDT
We're less than a week away from releasing 16.0 so wontfixing this on 15 since we certainly won't spin up a 15.0.2
Comment 15 Robert Kaiser (not working on stability any more) 2012-10-15 10:30:36 PDT
This is again rising on 16.0.1 compared to the betas, probably as release users are updating over to it - I'm pretty sure that the problem needs to be resolved on the Symantec side or at least in concert with them.
Comment 16 Alex Keybl [:akeybl] 2012-10-15 11:46:14 PDT
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #15)
> This is again rising on 16.0.1 compared to the betas, probably as release
> users are updating over to it - I'm pretty sure that the problem needs to be
> resolved on the Symantec side or at least in concert with them.

Does the affected DLL continue to be 2012.5.5.11? Norton let us know that this crash is fixed in a later version of their software. If not, please nominate for FF17 tracking.

Sadly, I don't think blocklisting the DLL is an option, since we typically stay away from AVG DLLs.
Comment 17 Jens S 2012-10-15 13:33:51 PDT
We believe we fixed this issue and a patch should be available by the end of the month.
Comment 18 Jorge Villalobos [:jorgev] 2012-11-01 15:04:21 PDT
(In reply to Jens S from comment #17)
> We believe we fixed this issue and a patch should be available by the end of
> the month.

Was the patch made available?
Comment 19 Scoobidiver (away) 2012-12-01 06:55:17 PST
With combined signatures, it's #7 top browser crasher w/o hangs in 17.0.
Comment 20 Robert Kaiser (not working on stability any more) 2012-12-13 14:17:44 PST
This is the #14 crash on 17.0.1 now.
Comment 21 Scoobidiver (away) 2013-01-09 01:39:21 PST
With combined signatures, it's #2 top browser crasher in the first hours of 18.0:
https://crash-stats.mozilla.com/report/list?signature=coffplgn.dll%400x3714
https://crash-stats.mozilla.com/report/list?signature=coffplgn.dll%400xc109

There are no correlations in 18.0 yet, but it still seems correlated to version 2012 and not 2013.
Comment 22 Scoobidiver (away) 2013-01-09 02:54:32 PST
Correlations are now available. Depending on the crash signature, versions 2012 or 2013 are affected:
 coffplgn.dll@0x3714|EXCEPTION_ACCESS_VIOLATION_EXEC (202 crashes)
    100% (202/202) vs.   1% (360/52744) coFFPlgn.dll
        100% (202/202) vs.   1% (266/52744) 2012.5.10.1
          0% (0/202) vs.   0% (13/52744) 2013.2.1.33
          0% (0/202) vs.   0% (81/52744) 2013.2.3.1

  coffplgn.dll@0xc109|EXCEPTION_ACCESS_VIOLATION_EXEC (48 crashes)
    100% (48/48) vs.   0% (92/52744) wincfi39.dll (3.9.1.0)
    100% (48/48) vs.   1% (360/52744) coFFPlgn.dll
          0% (0/48) vs.   1% (266/52744) 2012.5.10.1
          0% (0/48) vs.   0% (13/52744) 2013.2.1.33
        100% (48/48) vs.   0% (81/52744) 2013.2.3.1
Comment 23 Scoobidiver (away) 2013-01-11 03:01:04 PST
Norton Confidential 2012 crashes account for 6.6% of all crashes in 18.0.
Norton Confidential 2013 crashes account for 1.4% of all crashes in 18.0.
Comment 24 Tyler Downer [:Tyler] 2013-01-11 13:37:01 PST
Have we heard back form Norton on if they have pushed a patch, and if so, which version is fixed? Still getting reports of this in Firefox 18.
Comment 25 Alex Keybl [:akeybl] 2013-01-11 14:48:22 PST
(In reply to Tyler Downer [:Tyler] from comment #24)
> Have we heard back form Norton on if they have pushed a patch, and if so,
> which version is fixed? Still getting reports of this in Firefox 18.

They were planning on pushing a patch in the short term, but I'm reaching out to them to find out if the release date has changed.
Comment 27 Alex Keybl [:akeybl] 2013-01-13 14:25:27 PST
(In reply to Alex Keybl [:akeybl] from comment #25)
> (In reply to Tyler Downer [:Tyler] from comment #24)
> > Have we heard back form Norton on if they have pushed a patch, and if so,
> > which version is fixed? Still getting reports of this in Firefox 18.
> 
> They were planning on pushing a patch in the short term, but I'm reaching
> out to them to find out if the release date has changed.

We should have a fix for Norton 2013 soon, and Symantec is looking into the possibility of a fix for Norton 2012 as well.
Comment 28 Tyler Downer [:Tyler] 2013-01-15 08:07:24 PST
*** Bug 830107 has been marked as a duplicate of this bug. ***
Comment 29 Matthias Versen [:Matti] 2013-01-15 18:30:37 PST
*** Bug 831101 has been marked as a duplicate of this bug. ***
Comment 30 Scoobidiver (away) 2013-01-16 00:31:20 PST
Here are correlations per extensions version:
  coffplgn.dll@0x3714|EXCEPTION_ACCESS_VIOLATION_EXEC (7252 crashes)
    100% (7241/7252) vs.   7% (10967/152000) {2D3F3651-74B9-4795-BDEC-6DA2F431CB62}
        100% (7238/7252) vs.   6% (9524/152000) 2012.5.10.1
          0% (0/7252) vs.   0% (1/152000) 2012.5.7.2
          0% (3/7252) vs.   0% (3/152000) 2012.5.8.4
          0% (0/7252) vs.   0% (1/152000) 2013.2.1.36
          0% (0/7252) vs.   1% (1438/152000) 2013.2.3.1

  coffplgn.dll@0xc109|EXCEPTION_ACCESS_VIOLATION_EXEC (623 crashes)
    100% (623/623) vs.   7% (10967/152000) {2D3F3651-74B9-4795-BDEC-6DA2F431CB62}
          0% (0/623) vs.   6% (9524/152000) 2012.5.10.1
          0% (0/623) vs.   0% (1/152000) 2012.5.7.2
          0% (0/623) vs.   0% (3/152000) 2012.5.8.4
          0% (0/623) vs.   0% (1/152000) 2013.2.1.36
        100% (623/623) vs.   1% (1438/152000) 2013.2.3.1
Comment 31 Robert Kaiser (not working on stability any more) 2013-01-16 06:24:12 PST
(In reply to Scoobidiver from comment #30)
> Here are correlations per extensions version:

It's pretty clear that any of the signatures only corresponds to a single version, as changes in the binary will shift the address.

All that said, we are in communications with Norton and they're looking into the problems.

And here's a bit more complete picture of which signatures correspond to which coFFplgn.dll version:

Crash Signature      Crashes   Version
coffplgn.dll@0x3714    41334   2012.5.10.1
coffplgn.dll@0xc109     3695   2013.2.3.1
coffplgn.dll@0x3bbf6     861   2012.5.10.1
coffplgn.dll@0x3bc26     222   2012.5.8.4
coffplgn.dll@0x3bbfa      87   2012.5.10.1
coffplgn.dll@0x3bc2a      30   2012.5.8.4
...
coffplgn.dll@0xbf826       2   2013.2.3.1
...
coffplgn.dll@0xc7d5        1   2013.2.1.36

We have a number of low-volume signatures around, which I didn't list here - the latter two are just because I specifically looked for addresses that are similar to the 0xc109 one we're seeing for the topcrash in the 2013 version, just to find out if other 2013 variants would see similar crashes.
Comment 32 Alex Keybl [:akeybl] 2013-01-17 12:30:51 PST
QA - can you attempt to reproduce again given the STR & affected configs in bug 830107 and bug 831101?
Comment 33 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2013-01-17 14:23:06 PST
(In reply to Alex Keybl [:akeybl] from comment #32)
> QA - can you attempt to reproduce again given the STR & affected configs in
> bug 830107 and bug 831101?

I'll see what I can do following the Channel meeting today.
Comment 34 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2013-01-17 16:10:17 PST
I've been trying this for over an hour now and haven't stumbled upon a crash. Here are some details about what I've been doing so far...

I've installed some of the extensions common to the crash reports. I managed to find coffplgn.dll by installing Norton Internet Security 2012 Trial but it's not the right version (2012.1.0.30 instead of 20125.10.1). This is with all the latest Norton IS 2012 updates installed.

> {BBDA0591-3099-440a-AA10-41764D9DB4DB} 11.1.1.5 - 2
As near as I can tell, this is actually "Norton Vulnerability Protection 11.1.1.5 - 2" which is disabled by default in Firefox. I can't find any way to have this add-on enabled automatically without user intervention.

> {2D3F3651-74B9-4795-BDEC-6DA2F431CB62} 2012.5.10.1
As near as I can tell, this is actually "Norton Toolbar 2012.5.10.1" which again, is disabled by default. I have found no way to get this add-on enabled (there's no option to enabled in Add-ons Manager). Not even with the compatibility override (ie. Nightly Tester Tools).

Do the crash reports list add-ons that are installed and enabled, or do disabled add-ons show up in the report as well? If what the reports are showing is an indication that these add-ons are enabled then it might be worth finding out how.

By the way, here are my generalized testing steps:
1. Install Firefox 15 and start with a new profile
2. Install Norton Internet Security 2012, install any updates and restart
3. Do a pave-over install of Firefox 17.0 (needed to install some of the add-ons)
4. Install the following add-ons
* Nightly Tester Tools 3.3
* Noia Fox options 2.0.8 
* Toggle Persona 1.0.9
5. Restart Firefox and check for updates through the About dialog
6. Restart Firefox and browse the internet
7. Restart Firefox every once in a while to try to trigger a crash
8. Repeat the steps forcing a background update ping

I think Marcia was looking into this on one of the computers in the QA Lab as well. I'll let her comment if she found anything.
Comment 35 paul 2013-01-21 01:54:30 PST
I am the user who reported the issue and FYI it was suggested that I disable Norton Confidential (which I believe is Norton Identity Safe in NIS)> I have now done so and restarted Firefox, but the crashes continue, mainly when closing browser tabs, but I have also found the same issue if you try to "Start Private Browsing". I had not had any problems with version 17 and earlier, just v.18.
Comment 36 Scoobidiver (away) 2013-01-22 00:09:06 PST
*** Bug 833216 has been marked as a duplicate of this bug. ***
Comment 37 Robert Kaiser (not working on stability any more) 2013-01-22 06:13:34 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #31)
> coffplgn.dll@0xc7d5        1   2013.2.1.36

FYI, this one has been increasing in volume since - still about a factor 4 away from the 2013.2.3.1 signature, though.

All that said, the real bug concern remains to be the 2012 version.
Comment 38 Scoobidiver (away) 2013-01-27 01:58:29 PST
*** Bug 829929 has been marked as a duplicate of this bug. ***
Comment 39 Matthias Versen [:Matti] 2013-01-28 15:00:58 PST
*** Bug 835092 has been marked as a duplicate of this bug. ***
Comment 40 Scoobidiver (away) 2013-01-28 23:35:28 PST
A new Norton 2013 version was released that fixes those crashes:
http://community.norton.com/t5/Norton-Toolbar-Norton-Identity/Firefox-18-Support-for-Norton-Toolbar/m-p/900991#M8266
Comment 41 Scoobidiver (away) 2013-01-31 05:20:16 PST
Crashes fluctuate a lot from one day to the next:
Jan 21 12H - Jan 22 12H: 15,226
Jan 22 12H - Jan 23 12H: 14,656
Jan 23 12H - Jan 24 12H: 14,276
Jan 24 12H - Jan 25 12H:  8,921
Jan 25 12H - Jan 26 12H:  4,701
Jan 26 12H - Jan 27 12H:  4,400
Jan 27 12H - Jan 28 12H:  4,457
Jan 28 12H - Jan 29 12H:  8,229
Jan 29 12H - Jan 30 12H: 10,822
Jan 30 12H - Jan 31 12H:  9,440

Nevertheless, Norton 2013 crashes are fixed according to correlations per Norton version from today:
  coffplgn.dll@0x3715|EXCEPTION_ACCESS_VIOLATION_EXEC (8087 crashes)
    100% (8087/8087) vs.   6% (12078/204664) coFFPlgn.dll
        100% (8087/8087) vs.   5% (10819/204664) 2012.5.10.1

  couictlr.dll@0x2d280|EXCEPTION_ACCESS_VIOLATION_READ (753 crashes)
    100% (753/753) vs.   6% (12078/204664) coFFPlgn.dll
        100% (753/753) vs.   5% (10819/204664) 2012.5.10.1

  coffplgn.dll@0x3715|EXCEPTION_ACCESS_VIOLATION_READ (635 crashes)
    100% (635/635) vs.   6% (12078/204664) coFFPlgn.dll
        100% (635/635) vs.   5% (10819/204664) 2012.5.10.1

  xul.dll@0x10be2b4 | coffplgn.dll@0x3715|EXCEPTION_ACCESS_VIOLATION_EXEC (290 crashes)
    100% (290/290) vs.   6% (12078/204664) coFFPlgn.dll
        100% (290/290) vs.   5% (10819/204664) 2012.5.10.1

  coffplgn.dll@0xc7d6|EXCEPTION_ACCESS_VIOLATION_EXEC (286 crashes)
     97% (276/286) vs.   6% (12078/204664) coFFPlgn.dll
         96% (275/286) vs.   0% (489/204664) 2013.2.1.36
          0% (0/286) vs.   0% (158/204664) 2013.2.3.1
          0% (0/286) vs.   0% (590/204664) 2013.2.4.2

  coffplgn.dll@0x3715|EXCEPTION_ACCESS_VIOLATION_WRITE (275 crashes)
    100% (275/275) vs.   6% (12078/204664) coFFPlgn.dll
        100% (275/275) vs.   5% (10819/204664) 2012.5.10.1

  coffplgn.dll@0x3bbf6|EXCEPTION_ACCESS_VIOLATION_READ (127 crashes)
    100% (127/127) vs.   6% (12078/204664) coFFPlgn.dll
        100% (127/127) vs.   5% (10819/204664) 2012.5.10.1

  coffplgn.dll@0xc10a|EXCEPTION_ACCESS_VIOLATION_EXEC (72 crashes)
    100% (72/72) vs.   6% (12078/204664) coFFPlgn.dll
          0% (0/72) vs.   0% (489/204664) 2013.2.1.36
        100% (72/72) vs.   0% (158/204664) 2013.2.3.1
          0% (0/72) vs.   0% (590/204664) 2013.2.4.2
Comment 42 Robert Kaiser (not working on stability any more) 2013-01-31 05:48:11 PST
Yes, it looks like the coffplgn.dll@0xc109 signature (that somehow morphed into coffplgn.dll@0xc10a recently) from 2013.2.3.1 is gone, and @0xc7d6 is from 2013.2.1.36 which is even older, I guess. I haven't seen any prevalent crashes with 2013.2.4.2 so far, so things look good on the 2013 front.

The coffplgn.dll@0x3715 signature (morphed from what was coffplgn.dll@0x3714), which is the 2012.5.10.1 one, is still the major issue, but for some unknown reason also seems lower than it was before, maybe there was some push to get people over to the 2013 version. :)

We'll wait for the update to 2012 to ship though before we call this fixed. ;-)
Comment 43 Scoobidiver (away) 2013-01-31 15:09:26 PST
*** Bug 836763 has been marked as a duplicate of this bug. ***
Comment 44 Alex Keybl [:akeybl] 2013-02-04 07:46:15 PST
Removing from our tracking list as this won't require a fix on our side in the FF19 timeframe.
Comment 45 Scoobidiver (away) 2013-02-10 05:23:29 PST
A new crash signature in another Norton 2012 dll shows up as #24 top browser crasher: https://crash-stats.mozilla.com/report/list?signature=ccl110u.dll%400x61c04

Here are the latest correlations in 18.0.2:
  ccl110u.dll@0x61c04|EXCEPTION_ACCESS_VIOLATION_READ (402 crashes)
    100% (402/402) vs.   2% (2050/130050) coFFPlgn.dll
        100% (402/402) vs.   1% (768/130050) 2012.5.11.8

  coffplgn.dll@0x3715|EXCEPTION_ACCESS_VIOLATION_EXEC (374 crashes)
    100% (374/374) vs.   2% (2050/130050) coFFPlgn.dll
        100% (374/374) vs.   1% (684/130050) 2012.5.10.1 (previous version)

  couictlr.dll@0x2d280|EXCEPTION_ACCESS_VIOLATION_READ (126 crashes)
    100% (126/126) vs.   2% (2050/130050) coFFPlgn.dll
        100% (126/126) vs.   1% (684/130050) 2012.5.10.1 (previous version)

  coffplgn.dll@0x3af23|EXCEPTION_ACCESS_VIOLATION_READ (73 crashes)
    100% (73/73) vs.   2% (2050/130050) coFFPlgn.dll
        100% (73/73) vs.   1% (768/130050) 2012.5.11.8

  coffplgn.dll@0x3715|EXCEPTION_ACCESS_VIOLATION_READ (33 crashes)
    100% (33/33) vs.   2% (2050/130050) coFFPlgn.dll
        100% (33/33) vs.   1% (684/130050) 2012.5.10.1 (previous version)
Comment 46 Robert Kaiser (not working on stability any more) 2013-02-11 08:56:03 PST
(In reply to Scoobidiver from comment #45)
> A new crash signature in another Norton 2012 dll shows up as #24 top browser
> crasher:
> https://crash-stats.mozilla.com/report/list?signature=ccl110u.dll%400x61c04

We really should close up this bug and file new bugs for new issues that arise.
Comment 47 Scoobidiver (away) 2013-02-11 09:03:38 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #46)
> We really should close up this bug and file new bugs for new issues that
> arise.
Let's do that.

Note You need to log in before you can comment on or make changes to this bug.