Last Comment Bug 828184 - startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll (Websense Endpoint)
: startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll (Websense Endpoint)
Status: VERIFIED FIXED
[startupcrash][fixed in Websense Endp...
: crash, qawanted
Product: Core
Classification: Components
Component: General (show other bugs)
: 18 Branch
: x86 Windows 7
: -- critical with 1 vote (vote)
: mozilla21
Assigned To: Nobody; OK to take it and work on it
: juan becerra [:juanb]
Mentors:
: 865160 (view as bug list)
Depends on: 828296
Blocks:
  Show dependency treegraph
 
Reported: 2013-01-09 01:50 PST by Scoobidiver (away)
Modified: 2015-08-19 11:58 PDT (History)
18 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
-
fixed
-
fixed
-
verified
verified


Attachments
Block qipcap.dll, rev. 1 (1015 bytes, patch)
2013-01-15 07:55 PST, Benjamin Smedberg [:bsmedberg]
ehsan: review+
bajaj.bhavana: approval‑mozilla‑aurora+
bajaj.bhavana: approval‑mozilla‑beta+
bajaj.bhavana: approval‑mozilla‑release+
Details | Diff | Splinter Review

Description Scoobidiver (away) 2013-01-09 01:50:26 PST
It's #5 top browser crasher in the first hours of 18.0.
It's a startup crash but with not so many duplicates.

QIPCAP.dll (version 7.6) is a DLL that belongs to Websense Endpoint (https://www.websense.com/content/Home.aspx).

Signature 	nsPrefBranch::GetIntPref(char const*, int*) More Reports Search
UUID	de4a2771-9d31-474a-b784-df66f2130109
Date Processed	2013-01-09 09:22:19
Uptime	0
Last Crash	26.5 minutes before submission
Install Age	26.9 minutes since version was first installed.
Install Time	2013-01-09 08:54:58
Product	Firefox
Version	18.0
Build ID	20130104151925
Release Channel	release
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_WRITE
Crash Address	0x2
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x2a42, AdapterSubsysID: 024d1028, AdapterDriverVersion: 8.15.10.2302
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x2a42
Total Virtual Memory	2147352576
Available Virtual Memory	1933312000
System Memory Use Percentage	61
Available Page File	3456798720
Available Physical Memory	1414131712

Frame 	Module 	Signature 	Source
0 	xul.dll 	nsPrefBranch::GetIntPref 	modules/libpref/src/nsPrefBranch.cpp:181
1 	QIPCAP.dll 	QIPCAP.dll@0x1e840 	
2 	QIPCAP.dll 	QIPCAP.dll@0x1f726 	
3 	QIPCAP.dll 	QIPCAP.dll@0x214e4 	
4 	xul.dll 	nsObserverService::NotifyObservers 	xpcom/ds/nsObserverService.cpp:149
5 	xul.dll 	nsCOMPtr_base::~nsCOMPtr_base 	obj-firefox/dist/include/nsCOMPtr.h:408
6 	mozalloc.dll 	mozalloc.dll@0x109f 	
7 		@0xd 	
8 	mozglue.dll 	je_free 	memory/mozjemalloc/jemalloc.c:6589
9 	xul.dll 	nsLocalFile::Release 	xpcom/io/nsLocalFileWin.cpp:977
10 	xul.dll 	nsCOMPtr_base::~nsCOMPtr_base 	obj-firefox/dist/include/nsCOMPtr.h:408
11 	xul.dll 	NS_InitXPCOM2_P 	xpcom/build/nsXPComInit.cpp:447
12 	xul.dll 	ScopedXPCOMStartup::Initialize 	toolkit/xre/nsAppRunner.cpp:1181
13 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3935
14 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:105
15 	firefox.exe 	__tmainCRTStartup 	crtexe.c:552
16 	ntdll.dll 	__RtlUserThreadStart 	
17 	crypt32.dll 	ErrLog_LogError 	
18 	kernel32.dll 	WerpReportFaultInternal 	
19 	kernel32.dll 	WerpReportFaultInternal 	

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nsPrefBranch%3A%3AGetIntPref%28char+const*%2C+int*%29
Comment 1 Jorge Villalobos [:jorgev] 2013-01-09 07:22:34 PST
I've attempted to contact Websense about this.
Comment 2 bhavana bajaj [:bajaj] 2013-01-14 11:06:45 PST
CCing bsmedberg here to check if this could be related to https://bugzilla.mozilla.org/show_bug.cgi?id=828296 & recommendation regarding going ahead and blocklisting here.
Comment 3 Benjamin Smedberg [:bsmedberg] 2013-01-14 11:24:22 PST
Yes, this is very likely related to bug 828296, and the extension needs to be recompiled to take that into account.
Comment 4 Jorge Villalobos [:jorgev] 2013-01-14 12:32:43 PST
I don't see any extensions in the crash report. Maybe this is a DLL being injected into Firefox in some other way?
Comment 5 Benjamin Smedberg [:bsmedberg] 2013-01-14 12:53:05 PST
Indeed. But it's still using XPCOM APIs after it manages to get itself injected.
Comment 6 Tyler Downer [:Tyler] 2013-01-14 14:49:26 PST
This is starting to appear on SUMO. Any word from websense?
Comment 7 Jorge Villalobos [:jorgev] 2013-01-14 14:55:04 PST
No response yet. I'm adding the only contact on Bugzilla that seems to still be valid.

Note that this would need to be an in-product DLL block if we chose to go that way.
Comment 8 Alex Keybl [:akeybl] 2013-01-14 15:22:05 PST
(In reply to Jorge Villalobos [:jorgev] from comment #7)
> No response yet. I'm adding the only contact on Bugzilla that seems to still
> be valid.
> 
> Note that this would need to be an in-product DLL block if we chose to go
> that way.

I'm also trying to give WebSense a call right now. I want to verify that they either plan to fix the issue in the short term for 7.6 users, or what the user experience will be like if we block the DLL.

(In reply to Tyler Downer [:Tyler] from comment #6)
> This is starting to appear on SUMO.

For reference: https://support.mozilla.org/en-US/questions/946986?esab=a&s=websense&r=1&as=s
And crash comments: https://crash-stats.mozilla.com/report/list?range_value=7&range_unit=days&date=2013-01-14&signature=nsPrefBranch%3A%3AGetIntPref%28char%20const*%2C%20int*%29&version=Firefox%3A18.0#comments
Comment 9 Alex Keybl [:akeybl] 2013-01-14 15:58:03 PST
No dice on the phone. Somebody will call me back, but the technical support people need to be contacted by customers. We need to grab some emails to perform outreach, so that their IT can contact Websense.

Is there any evidence that v7.6 of QIPCAP.dll is working for some users? If not, we should go ahead and blocklist.
Comment 10 Alex Keybl [:akeybl] 2013-01-14 17:49:42 PST
Marcia helped with the emails, since she's PT.
Comment 11 Benjamin Smedberg [:bsmedberg] 2013-01-14 18:00:13 PST
Note that the block would have to be in-product, given that this is a DLL block.
Comment 12 Benjamin Smedberg [:bsmedberg] 2013-01-15 07:55:31 PST
Created attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1
Comment 13 Robert Kaiser 2013-01-15 08:49:23 PST
(In reply to Alex Keybl [:akeybl] from comment #9)
> Is there any evidence that v7.6 of QIPCAP.dll is working for some users?

I have no idea how I should find out about that.
Comment 14 Scoobidiver (away) 2013-01-15 09:09:25 PST
It happens with recent QIPCAP.dll versions, including the latest one:
  nsPrefBranch::GetIntPref(char const*, int*)|EXCEPTION_ACCESS_VIOLATION_WRITE (347 crashes)
    100% (347/347) vs.   0% (352/152000) QIPCAP.dll
          0% (0/347) vs.   0% (1/152000) 7.6.808.1
          0% (0/347) vs.   0% (1/152000) 7.6.811.1
          3% (9/347) vs.   0% (11/152000) 7.6.812.1
          3% (10/347) vs.   0% (11/152000) 7.6.813.1
         95% (328/347) vs.   0% (328/152000) 7.6.815.1
Comment 15 :Ehsan Akhgari 2013-01-15 10:55:02 PST
Comment on attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1

Review of attachment 702323 [details] [diff] [review]:
-----------------------------------------------------------------

r=me
Comment 17 :Ehsan Akhgari 2013-01-15 11:11:38 PST
Comment on attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1

This patch is not risky and doesn't change strings or uuids.
Comment 18 Benjamin Smedberg [:bsmedberg] 2013-01-15 11:14:28 PST
Comment on attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1

There's no reason to take this for the ESRs because the crash was caused by an interface change only in FF18.
Comment 19 bhavana bajaj [:bajaj] 2013-01-15 11:16:47 PST
Adding qawanted to help with verification that blocklist works as expected.
Comment 20 Benjamin Smedberg [:bsmedberg] 2013-01-15 11:18:00 PST
I don't expect the crash to be reproduceable on nightly because we fixed the nsIPrefBranch IID.
Comment 21 Scoobidiver (away) 2013-01-15 11:36:02 PST
Should we land this patch only in Release and Beta, and not in Aurora and m-c where bug 828296 is fixed?
Comment 22 Benjamin Smedberg [:bsmedberg] 2013-01-15 11:40:15 PST
The DLL is very likely not going to work in m-c/aurora, so I don't think there's harm in blocking it there also, and not blocking it would create a weird situation where it's loaded into a later version but not an earlier version.
Comment 24 Ed Morley [:emorley] 2013-01-16 12:38:03 PST
https://hg.mozilla.org/mozilla-central/rev/1fecb291afcc
Comment 25 juan becerra [:juanb] 2013-01-16 23:20:49 PST
I tried to find Websense Endpoint in their product list, but after registering and getting a trial account I'm still waiting for their suite to finish downloading (2gb+).

It looks like it is a content filtering system, and most search results were about bypassing it through a proxy. I'll look into it tomorrow when it finishes downloading.
Comment 26 juan becerra [:juanb] 2013-01-17 12:10:48 PST
I installed the web security part of the test suite, but after installing there was no process loading the qipcap.dll (as show using the program ListDlls). Firefox 18.0 didn't load it, and as far as I can't tell, it wasn't even found in the installation I performed.

If anyone has any ideas, please let me know.
Comment 27 netpq 2013-01-17 13:43:20 PST
You don't need to fix because Websense did. FF 17& 18 dumped nsIPrefBranch2 interface and changed it's ID.
Comment 28 Scoobidiver (away) 2013-01-25 23:46:14 PST
It's still not fixed by the DLL blocklist:
  nsPrefBranch::GetIntPref(char const*, int*)|EXCEPTION_ACCESS_VIOLATION_WRITE (157 crashes)
    100% (157/157) vs.   0% (168/137016) QIPCAP.dll
          0% (0/157) vs.   0% (1/137016) 7.3.600.0
          1% (1/157) vs.   0% (6/137016) 7.6.812.1
          1% (1/157) vs.   0% (6/137016) 7.6.813.1
         99% (155/157) vs.   0% (155/137016) 7.6.815.1

(In reply to netpq from comment #27)
> You don't need to fix because Websense did. FF 17& 18 dumped nsIPrefBranch2
> interface and changed it's ID.
Where did you get this info?
Comment 29 Robert Kaiser 2013-01-26 08:00:10 PST
> (In reply to netpq from comment #27)
> > You don't need to fix because Websense did. FF 17& 18 dumped nsIPrefBranch2
> > interface and changed it's ID.
> Where did you get this info?

See bug 828296 - this is one of the more common causes of crashes with third-party binary software in Firefox 18. FYI, we suspect that the Norton Confidential crashes spiking in this release is connected to that as well, see the dependencies of that bug (this here is one of them, BTW).
Comment 31 Benjamin Smedberg [:bsmedberg] 2013-01-28 13:12:18 PST
In that case it would be nice to figure out how the DLL is getting loaded and past our blocklist, but I don't plan on looking at this further.
Comment 32 Alex Keybl [:akeybl] 2013-01-29 11:56:40 PST
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #31)
> In that case it would be nice to figure out how the DLL is getting loaded
> and past our blocklist, but I don't plan on looking at this further.

Users have contacted Websense at this point and the crash remains quite low. I agree that we shouldn't devote more time to this issue specifically.
Comment 33 bugzilla 2013-01-29 13:23:53 PST
low crash depending of the organization, for us is a serious problem with my 700 users were most of them use Firefox and we must roll out websense to all the users, it crash on all Firefox versions > 17.x , tested in the beta versions as well.

Do we have to recomend stop using Firefox including dev teams that produce web portals for customers and let the customers know that Firefox is no supported ?

A bit ridiculous..
Comment 35 Ioana (away) 2013-02-13 02:00:11 PST
There are almost 30 crashes on Firefox 19 beta 5, which only got released a week ago:
https://crash-stats.mozilla.com/report/list?query_search=signature&query_type=contains&reason_type=contains&range_value=4&range_unit=weeks&hang_type=any&process_type=any&signature=nsPrefBranch%3A%3AGetIntPref%28char%20const%2A%2C%20int%2A%29

I've only seen 2 post-fix crashes on Firefox 20 though, and none on Firefox 21.
Comment 36 Scoobidiver (away) 2013-02-13 02:39:43 PST
(In reply to Ioana Budnar [QA] from comment #35)
> There are almost 30 crashes on Firefox 19 beta 5, which only got released a
> week ago:
The fix is not the DLL blocklist but the Websense Endpoint update so those who crash haven't updated it.
Comment 37 bugzilla 2013-02-18 13:24:08 PST
Just in case helps to someone,

I have been doing lots of testing latelly with Firefox 18.02 and different versions of websense enpoint client, the issue with the version 18.x seems to be resolved with websense endpoint version 1138 I just got today, unfortunatelly it is a preproduction version and the latest production version is 1130 to wich Firefox 18.x does not run therefore you need to request the version 1138 from websense support teams.
Comment 38 Ioana (away) 2013-02-22 05:52:16 PST
Considering the number of crashes in the last 4 weeks (5 on Fx 20, 0 on Fx 21 and 22), we can consider this bug fixed and verified.
Comment 39 Scoobidiver (away) 2013-04-24 22:44:19 PDT
*** Bug 865160 has been marked as a duplicate of this bug. ***
Comment 40 Swarnava Sengupta (:Swarnava) 2013-04-29 01:58:02 PDT
Firefox 21 beta crash id bp-92f342ba-bcfd-4ad0-9daf-c609a2130429
Comment 41 Scoobidiver (away) 2013-04-29 02:09:36 PDT
(In reply to Swarnava Sengupta (:Swarnava) from comment #40)
> Firefox 21 beta crash id bp-92f342ba-bcfd-4ad0-9daf-c609a2130429
You are using a Websense Endpoint version that doesn't contain the fix (see the whiteboard).
Comment 42 Swarnava Sengupta (:Swarnava) 2013-04-29 06:39:53 PDT
oh yes, i just told that user to update websense!
Comment 43 Paul Pietromonaco 2013-05-22 14:21:37 PDT
Sorry for jumping in here, but I believe this is the correct location for this comment.  At work, we are running Websense Endpoint 1139.  The current version of Firefox - 21.xx - works fine, but the latest ESR version of Firefox - 17.0.6 - crashes upon startup.  If I disable Websense, ESR works normally.  There may not be any action required here - just mentioning that this is still affecting current ESR builds.
Comment 44 Scoobidiver (away) 2013-05-22 14:26:46 PDT
(In reply to Paul Pietromonaco from comment #43)
> There may not be any action required here - just mentioning that this is still 
> affecting current ESR builds.
An extension with binary can't be compatible with 17.0 ESR and the current version if the binary has been changed after 17.0. Usually, ESR users don't need extensions.
Comment 45 motospam 2013-11-04 13:52:08 PST Comment hidden (abuse)

Note You need to log in before you can comment on or make changes to this bug.