Closed Bug 738661 Opened 13 years ago Closed 6 years ago

crash in nmsvc from CovenantEyes

Categories

(Core :: Networking, defect, P3)

Product:
Core
Component:
Networking
Version:
12 Branch
Platform:
x86
Windows 7
Type:
defect

Tracking

()

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox12 - ---
firefox19 - ---
firefox47 --- wontfix
firefox48 --- wontfix
firefox49 --- wontfix
firefox-esr45 --- wontfix

People

(Reporter: scoobidiver, Unassigned)

Details

(Keywords: crash, reproducible, Whiteboard: [tbird crash][necko-backlog])

Crash Data

With combined signatures, it's #23 top browser crasher in 12.0b1.

Signature 	nmsvc.dll@0xeffe More Reports Search
UUID	4ab03692-a0c0-4269-ae4a-55c222120322
Date Processed	2012-03-22 02:59:42
Uptime	32
Last Crash	35 seconds before submission
Install Age	4.0 days since version was first installed.
Install Time	2012-03-18 02:02:57
Product	Firefox
Version	12.0
Build ID	20120314195616
Release Channel	beta
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	x86
Build Architecture Info	AuthenticAMD family 20 model 2 stepping 0
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x15af9000
User Comments	why the heck does it keep crashing. 20 to 30 times a day.
App Notes 	
AdapterVendorID: 0x1002, AdapterDeviceID: 0x9807, AdapterSubsysID: 05981025, AdapterDriverVersion: 8.861.0.0
D2D? D2D+ DWrite? DWrite+ D3D10 Layers? D3D10 Layers+ 
Processor Notes	
EMCheckCompatibility	True
Total Virtual Memory	4294836224
Available Virtual Memory	3597475840
System Memory Use Percentage	38
Available Page File	6241071104
Available Physical Memory	2443124736

Frame 	Module 	Signature [Expand] 	Source
0 	nmSvc.dll 	nmSvc.dll@0xeffe 	
1 	nmSvc.dll 	nmSvc.dll@0x8344 	
2 	nmSvc.dll 	nmSvc.dll@0xe199 	
3 	nmSvc.dll 	nmSvc.dll@0xea09 	
4 	nmSvc.dll 	nmSvc.dll@0x2798 	
5 	CESpy.dll 	CESpy.dll@0xc63d 	
6 	ws2_32.dll 	WSARecv 	
7 	wsock32.dll 	recv 	
8 	nspr4.dll 	SocketRead 	nsprpub/pr/src/io/prsocket.c:657
9 	xul.dll 	nsSocketInputStream::Read 	netwerk/base/src/nsSocketTransport2.cpp:362
10 	xul.dll 	nsHttpConnection::OnWriteSegment 	netwerk/protocol/http/nsHttpConnection.cpp:1052
11 	xul.dll 	nsHttpTransaction::WritePipeSegment 	netwerk/protocol/http/nsHttpTransaction.cpp:544
12 		@0x7fff 	
13 	xul.dll 	nsHttpTransaction::WriteSegments 	netwerk/protocol/http/nsHttpTransaction.cpp:574
14 	xul.dll 	nsCSSFrameConstructor::GetRangeInsertionPoint 	layout/base/nsCSSFrameConstructor.cpp:6441
15 	xul.dll 	nsHttpConnection::OnSocketReadable 	netwerk/protocol/http/nsHttpConnection.cpp:1084
16 	xul.dll 	nsHttpConnection::OnInputStreamReady 	netwerk/protocol/http/nsHttpConnection.cpp:1204
17 	xul.dll 	nsSocketInputStream::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:255
18 	xul.dll 	nsSocketTransport::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:1574
19 	xul.dll 	nsSocketTransportService::DoPollIteration 	netwerk/base/src/nsSocketTransportService2.cpp:759
20 	xul.dll 	nsSocketTransportService::Run 	netwerk/base/src/nsSocketTransportService2.cpp:642
21 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:657
22 	xul.dll 	nsThreadStartupEvent::Run 	xpcom/threads/nsThread.cpp:218
23 	nspr4.dll 	_PR_NativeRunThread 	nsprpub/pr/src/threads/combined/pruthr.c:426
24 	nspr4.dll 	pr_root 	nsprpub/pr/src/md/windows/w95thred.c:122
25 	msvcr80.dll 	_callthreadstartex 	f:\\dd\\vctools\\crt_bld\\self_x86\\crt\\src\\threadex.c:348
26 	msvcr80.dll 	_SEH_epilog4 	
27 	msvcr80.dll 	_threadstartex 	f:\\dd\\vctools\\crt_bld\\self_x86\\crt\\src\\threadex.c:326
28 	kernel32.dll 	BaseThreadInitThunk 	
29 	ntdll.dll 	__RtlUserThreadStart 	
30 	ntdll.dll 	_RtlUserThreadStart 

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nmsvc.dll%400xeffe
https://crash-stats.mozilla.com/report/list?signature=nmsvc.dll%400xffbe
Email sent today to info@CovenantEyes.com advising them of the crash issue.
We should get some QAing around this to help CovenantEyes reproduce the issue, or to understand if we've done anything to tickle this crash.
tracking-firefox12: ? → +
Keywords: qawanted
As part of my inquiry in Comment 1, I asked for a test account since they don't seem to have any trial accounts according to: http://www.covenanteyes.com/services/pricing/
I received an email back from them this evening with a promo code. I will create an account tomorrow and report back.
I created a promo account using Accountability Monthly filter. They have three choices of filters. I installed it and have been running FF 12 B2 but so far have not generated any crashes.
I have now tried different testing scenarios on two different Win 7 machines with Covenant eyes installed running 12b2 and so far have not generated any crashes.  I also tested with some of the addons found in the manual reports (https://crash-analysis.mozilla.com/crash_analysis/20120328/20120328_Firefox_12.0-interesting-addons.txt.gz).

If you look at the uptime data for both signatures there is a bit of a pattern if a longer uptime followed by a shorter uptime. Maybe I have to wait this crash out a bit.

I guess the next step would be try reaching out to Covenant eyes to see if anyone there has been able to reproduce it. I can try responding to the customer service rep that emailed me and see if there is a dev contact there we could work with.
(In reply to Marcia Knous [:marcia] from comment #6)
> I can try responding to the customer service rep that emailed me
> and see if there is a dev contact there we could work with.

That would be great. Please do so.
Mail sent requesting a dev contact on their side.
I did hit this crash on my Win 7 Sony Vaio today, https://crash-stats.mozilla.com/report/index/bp-0e957831-a21c-4d7b-b9fe-efb862120329.

At the time of the crash I had just clicked a link to download Adobe Flash from http://get.adobe.com/flashplayer/?promoid=BUIGP. Let's see if I can try to more reliably reproduce it.
Any luck reproducing this, Marcia?
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #10)
> Any luck reproducing this, Marcia?

Sorry, just noticed the timestamp was from today, not yesterday. I'm not trying to rush you :)
Alex and I talked about getting this in a record and replay to see if there is a way to get any more information. I have to find out if bent still has a machine set up equipped to do this.

During my testing I managed to repro another time but there doesn't seem to be a pattern to the crashes.
Okay, I think I have better STR here after playing around with this a bit more.

STR:
1. Log into Covenant Eyes
2. Load netflix.com.
3. Covenant eyes tries to block the URL so I enter my Covenant eyes user name and password and allow the site to load
4. I enter my Netflix user name and password and then I crash.
5. Subsequent visits to the Netflix site result in a crash.

So it would seen from this testing that after I have "allowed" a site that Covenant Eyes has blocked, subsequent visits to that site seem to result in a crash.
Keywords: reproducible
Now that we have a reproducible testcase, who is the right person on the dev side to get involved?
Assignee: nobody → bsmith
Brian: I have the machine in the Mountain View office that is crashing if you need to borrow it.
(In reply to Marcia Knous [:marcia] from comment #15)
> Brian: I have the machine in the Mountain View office that is crashing if
> you need to borrow it.

Given the above STR, can we help Brian find a regression window?
Keywords: regressionwindow-wanted
Brian requested I try to reproduce the crash using Firefox 11 - using Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0 I have not been able to reproduce the crash that I am hitting consistently with the 12 beta on the same machine.

I can check to make sure that this started in the first 12 beta since I think the build I am testing with is Beta 2.
(In reply to Marcia Knous [:marcia] from comment #17)
> Brian requested I try to reproduce the crash using Firefox 11 - using
> Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0 I
> have not been able to reproduce the crash that I am hitting consistently
> with the 12 beta on the same machine.
> 
> I can check to make sure that this started in the first 12 beta since I
> think the build I am testing with is Beta 2.

Marcia/Anthony - in the channel meeting we discussed finding a smaller regression window than some time in FF12. Can we get that for Brian as well?
Marcia had told me a couple of days ago that she was able to trace the regression down between 11.0-final and 12.0b1. I advised her to check the mozilla-central 12.0a1 Nightlies to try to track the regression range. Unfortunately I do not have access to the hardware so I cannot assist.

Marcia, were you able to make any progress on this?
The Sony Vaio I was testing on blue screened, so as soon as can get that machine up and running I will retest this.
Turns out hunting down an exact regression range for this bug will be a bit difficult due to the fact that sometimes the browser does not crash right away when I add the filter. I also have to spend time finding sites which will invoke the Covenant eyes filter in order to apply it.

During my testing I still have never been able to crash using Firefox 11. Using the same profile, Firefox 12 latest beta seems to crash fairly easily and if I try to session restore then the browser continually will crash.

I started looking at builds back to 20111220 and so far the only one that has crashed in the range between Firefox 11 and 12 is 20120115.
What is the best way to query all the changes that were made between Firefox 11 RTM and the 20120115 build?
(In reply to Brian Smith (:bsmith) from comment #22)
> What is the best way to query all the changes that were made between Firefox
> 11 RTM and the 20120115 build?

https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2011-12-20&enddate=2012-01-15

or

hg log --date 2011-12-20 to 2012-01-15

should do the trick. Would it make sense to hook up the crashing case to a debugger?
This is no longer a top crash for recent FF12 betas - Brian will still look into this in case it explodes post-release in the same way it did after FF12 landed on beta. We're specifically looking to figure out where the bug lies (Firefox or CovenantEyes).

I've also asked Kev if we have any contacts at CovenantEyes that Brian can work with, since he strongly suspects this is a 3rd party issue after performing code inspection.
Keywords: topcrash
I have had a ton of crashes lately after upgrading to FF12. Any ETA on a solution to this?
I am experiencing spontaneous crashes of FF12.  It will crash while nothing is being done on the computer so I don't think it is the same 'bug' as this but I know how you hate new bugs being reported when there are others already reporting them so I'm posting here first.  I think 'Wes' and I might be experiencing the same thing.

I have followed the suggestions and updated both Windows(XP Pro) and also run in 'safe mode' with no improvement.  Please help as this is more than annoying.

Thanks,
Rob
Wes and Rob, you should disable ConvenantEyes and contact its support site to get it fixed.
The only solution Mozilla could do is to blocklist this DLL so that it disables ConvenantEyes for every users.
I'm pretty sure my problem has nothing to do with 'CoventEyes' as I don't know what that is and I'm pretty sure I don't have it.  I guess this is not the correct place for my post but it was the most recent and Wes's comment sounded more like my issue.

Sorry for the confusion.
Rob, can you provide one of your crashID from about:crashes? Can you provide the location of nmSvc.dll on your computer?
Thanks for your help on this.  I'm learning as I go here.  It seems this is clearly  a different issue: 
https://crash-stats.mozilla.com/report/index/7b7d3139-ba01-45a0-8616-ed1782120428
https://crash-stats.mozilla.com/report/index/53f5191d-1937-44e7-b441-188712120429
https://crash-stats.mozilla.com/report/index/59564c93-2708-45cf-ad0e-59b1e2120429

Just did a search of the entire machine, hidden and system files included and there is no nmSvc.dll to be found.

I guess this should be in it's own thread.  Can someone make that happen or do I need to start a new one?
Looking at your crash reports, it seems there is a bug on file already which is correlated to the crash signature you are seeing:

 * Bug 714761 - Firefox 12.0a1 Crash Report [@ JSObject::getGeneric(JSContext*, JSObject*, int, JS::Value*) ]

Feel free to comment on that bug report.
Thanks so much and sorry for posting in the wrong place.  Now that I know where to look for crash reports I should have better luck finding the right threads to post in.

Thanks again!
This signature has around ~1300 crashes in the last week across all versions, including some crashes in beta and aurora. The lion's share of crashes affect users that are using FF 12.
also seen in thunderbird per https://getsatisfaction.com/mozilla_messaging/topics/keeps_on_crashing-l3d9t
Whiteboard: [tbird crash]
I am seeing this problem every few seconds in FFX 13:

http://crash-stats.mozilla.com/report/index/bp-bf3ea54f-072d-4d47-9a48-446852120607
http://crash-stats.mozilla.com/report/index/bp-4e346b24-6a2d-437b-a2b0-6e5d32120607
http://crash-stats.mozilla.com/report/index/bp-a994cef4-8519-4970-ac5c-760d82120607

I'm sure there will be lots more CE users reporting crashes with the new version.

Happy to try anything people have suggestions for...
(In reply to Marcia Knous [:marcia] from comment #13)
> Okay, I think I have better STR here after playing around with this a bit
> more.
> 
> STR:
> 1. Log into Covenant Eyes
> 2. Load netflix.com.
> 3. Covenant eyes tries to block the URL so I enter my Covenant eyes user
> name and password and allow the site to load
> 4. I enter my Netflix user name and password and then I crash.
> 5. Subsequent visits to the Netflix site result in a crash.
> 
> So it would seen from this testing that after I have "allowed" a site that
> Covenant Eyes has blocked, subsequent visits to that site seem to result in
> a crash.

I'm coming to this party a little late, but this description doesn't seem to make sense. CE doesn't block *any* websites. It's either logged in or it's not. If you're logged in, you can browse normally. CE just records URLs and may choose to report them to your "buddy" if it finds them suspicious. 

I'm seeing lots of crashes with upgrade to FFX 13.
With combined signatures, it's #36 top browser crasher in 12.0 and #35 in 13.0, so pretty stable.
Keywords: topcrash
Assignee: bsmith → nobody
It's a low volume crash in 15.0.1.
Keywords: topcrash
I find that it often crashes when going to a yahoo news page from the news shown with Yahoo mail. I originally thought it was from Flash.
(In reply to Aharon from comment #39)
> I find that it often crashes when going to a yahoo news page from the news
> shown with Yahoo mail. I originally thought it was from Flash.

Would you be willing to do some regression testing for us? We've been unable to reproduce this internally after several months of trying. I can provide you further instruction if you are willing.
Keywords: qawanted
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #40)
>  
> Would you be willing to do some regression testing for us? We've been unable
> to reproduce this internally after several months of trying. I can provide
> you further instruction if you are willing.
After feeling like 14/15 were a little better, I get this crash almost every few minutes on my home machine in 17 - I suspect it's related to my slow home connection (~300kbit, maybe less) because I don't see it often at work.

I would be willing to do testing.

Heck I don't know what Mozilla is building with but I would be happy to run under a debugger. Have the CE people offered to provide a version of the DLL with debugging symbols?
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #40)
> (In reply to Aharon from comment #39)
> > I find that it often crashes when going to a yahoo news page from the news
> > shown with Yahoo mail. I originally thought it was from Flash.
> 
> Would you be willing to do some regression testing for us? We've been unable
> to reproduce this internally after several months of trying. I can provide
> you further instruction if you are willing.

I fortunately have not had this crash in a little while. Maybe due to an update for Firefox and/or CE. It also coincided somewhat with my getting a faster connection that it stopped happening. Maybe that has to do with it. This would back up Ari's suspicion. I therefore don't think that I would be of much help at this point.
Ari, this bug was originally filed against Firefox 12, so I would expect the crashes to start happening there. The best way to confirm that would be to try using Firefox 11 and 12 and doing a comparison between the two. If you can crash Firefox 12 but not Firefox 11, it means the bug probably started in Firefox 12 Nightly at some point.

Fx11: ftp://ftp.mozilla.org/pub/firefox/releases/11.0/
Fx12: ftp://ftp.mozilla.org/pub/firefox/releases/12.0/

Thanks
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #43)
> The best way to confirm that would be to try using Firefox 11 and 12 and doing 
> a comparison between the two. 

Okey-doke, I'll roll on back and see if things improve.

If I narrow it down to between 11 and 12, is the next step a binary search thru the nightlies?
(In reply to Ari Heitner from comment #44)
> Okey-doke, I'll roll on back and see if things improve.

Thanks!

> If I narrow it down to between 11 and 12, is the next step a binary search
> thru the nightlies?

Right. There's an app for that:
http://mozilla.github.com/mozregression/
(In reply to Ari Heitner from comment #44)
> (In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #43)
> > The best way to confirm that would be to try using Firefox 11 and 12 and doing 
> > a comparison between the two. 
> If I narrow it down to between 11 and 12 [...]

Guess what folks? 11.0 crashes plenty too.

Continuing on back to 6.0, I guess from there I'll go back to 3.5 or so if necessary. If it still crashes, does everyone agree it's a CE change that caused it?

To that end - would whoever got in touch with the CE people care to share their contact info by private email to me?
(In reply to Ari Heitner from comment #46)
> (In reply to Ari Heitner from comment #44)
> > (In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #43)
> > > The best way to confirm that would be to try using Firefox 11 and 12 and doing 
> > > a comparison between the two. 
> > If I narrow it down to between 11 and 12 [...]
> 
> Guess what folks? 11.0 crashes plenty too.
> 
> Continuing on back to 6.0

Thanks Ari. If it goes back as far as Firefox 4.0 then that pretty much invalidates the assumption that this is a Firefox regression. There's not much point in going back any further than that.

> To that end - would whoever got in touch with the CE people care to share
> their contact info by private email to me?

If it's not a regression we will have to consider workarounds (traditionally outreach and then blocklisting in worse-case scenario). However, please focus on seeing if this goes back as far as Firefox 4. Thank you.
Crash Signature: [@ nmsvc.dll@0xeffe] [@ nmsvc.dll@0xffbe] [@ nmsvc.dll@0xfa8e] [@ nmsvc.dll@0xff2e] → [@ nmsvc.dll@0xeffe] [@ nmsvc.dll@0xffbe] [@ nmsvc.dll@0xfa8e] [@ nmsvc.dll@0xff2e] [@ nmsvc.dll@0x10527b]
One thing that I am still finding with Firefox 17 is that every now and then pages stop loading for no reasonuntil I close Firefox. Once I close it CovenantEyesHelper.exe starts using ~25%cpu+ and nothing can load from the internet until I restart CE. This happens with Opera as well, but not Chrome. Every time I close FF I get high CPU from CE.
Official results are in for the new year: 6.0 crashes too.

Will go back to 4.0 and see what happens.
Thanks Ari, let us know how you make out with Firefox 4.

Scoobidiver, could you please check crash-stats to see where this sits in the topcrash landscape for Firefox 18?
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #50)
> Scoobidiver, could you please check crash-stats to see where this sits in
> the topcrash landscape for Firefox 18?
It's #222 top browser crasher in 17.0.1, #254 in 18.0b7 and #985 in 19.0a2.
Given Scoobidiver's analysis, no need to track for upcoming releases. Looks like people have stopped using Firefox/CovenantEyes together.
tracking-firefox19: ? → -
Repeated crashes related to this bug.

Firefox started crashing consistently after the last update and it seems to be related to CovenantEyes. Websites it commonly occurs on: www.grooveshark.com and
http://www.investing.com/currencies/usd-cad-chart

Some of the latest crash IDs:
bp-9d35083f-cedc-49c1-a2d7-6d24e2130227
bp-040d7996-a004-445b-b28c-bd87a2130227
bp-a50d9fff-ef5e-4b8f-88c9-abcc32130227
bp-78b770df-5d2d-42d9-8d04-aad702130227
bp-756a1b06-ab20-4446-92e9-7bd2c2130227
(In reply to gbruintjes from comment #53)
> Repeated crashes related to this bug.

We've had no success reproducing this internally over the last year. Would you be willing to try to find a regression range for this bug using the nightlies? I can provide guidance as needed.
I had no success getting a regression range. I went back to 6.0 or something ridiculous, it all crashed. Latest also crashes.
(In reply to Ari Heitner from comment #55)
> I had no success getting a regression range. I went back to 6.0 or something
> ridiculous, it all crashed. Latest also crashes.

In that case this is more likely a bug in Covenant Eyes than it is a bug in Firefox. If we have no hope of success through engaging with Covenant Eyes and can't blocklist the offending DLL from loading, can we at least get a support article posted instructing users how to work around this issue (ie. removing Covenant Eyes or switching to a different browser)?

Additionally, I don't see much point in keeping this bug in the NEW state if it's unreasonable to expect that we'll ever fix this.
(In reply to Tyler Downer [:Tyler] from comment #57)
> Communicated to our SUMO community at
> https://support.mozilla.org/en-US/forums/contributors/709022

Thanks Tyler.
For the record, my limited experience indicates that other browsers crash just fine too. Crash may be linked to a race condition - seems to happen on slower machines.
this remains an issue with firefox too
Whiteboard: [tbird crash] → [tbird crash][necko-backlog]
Crash volume for signature 'nmsvc.dll@0x10527b':
 - nightly (version 50): 0 crash from 2016-06-06.
 - aurora  (version 49): 0 crash from 2016-06-07.
 - beta    (version 48): 16 crashes from 2016-06-06.
 - release (version 47): 379 crashes from 2016-05-31.
 - esr     (version 45): 17 crashes from 2016-04-07.

Crash volume on the last weeks:
             Week N-1   Week N-2   Week N-3   Week N-4   Week N-5   Week N-6   Week N-7
 - nightly          0          0          0          0          0          0          0
 - aurora           0          0          0          0          0          0          0
 - beta             3          2          3          2          3          1          1
 - release         53         56         57         59         65         60         11
 - esr              3          2          1          2          3          2          1

Affected platform: Windows
status-firefox47: --- → affected
status-firefox48: --- → affected
status-firefox-esr45: --- → affected
Crash volume for signature 'nmsvc.dll@0x10527b':
 - nightly (version 51): 0 crashes from 2016-08-01.
 - aurora  (version 50): 0 crashes from 2016-08-01.
 - beta    (version 49): 16 crashes from 2016-08-02.
 - release (version 48): 51 crashes from 2016-07-25.
 - esr     (version 45): 28 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       0       0       0
 - aurora        0       0       0
 - beta          5       7       2
 - release      14      13       5
 - esr           0       0      10

Affected platform: Windows

Crash rank on the last 7 days:
           Browser     Content   Plugin
 - nightly
 - aurora
 - beta    #2725
 - release #916
 - esr
status-firefox49: --- → affected
Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258
Priority: -- → P1
Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258
Priority: P1 → P3

Closing because no crashes reported for 12 weeks.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.