828184 - startup crash in nsPrefBranch::GetIntPref with QIPCAP.dll (Websense Endpoint)

Status: VERIFIED FIXED

(Core :: General, defect)

Description

[Scoobidiver (away)]

It's #5 top browser crasher in the first hours of 18.0.
It's a startup crash but with not so many duplicates.

QIPCAP.dll (version 7.6) is a DLL that belongs to Websense Endpoint (https://www.websense.com/content/Home.aspx).

Signature 	nsPrefBranch::GetIntPref(char const*, int*) More Reports Search
UUID	de4a2771-9d31-474a-b784-df66f2130109
Date Processed	2013-01-09 09:22:19
Uptime	0
Last Crash	26.5 minutes before submission
Install Age	26.9 minutes since version was first installed.
Install Time	2013-01-09 08:54:58
Product	Firefox
Version	18.0
Build ID	20130104151925
Release Channel	release
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_WRITE
Crash Address	0x2
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x2a42, AdapterSubsysID: 024d1028, AdapterDriverVersion: 8.15.10.2302
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x2a42
Total Virtual Memory	2147352576
Available Virtual Memory	1933312000
System Memory Use Percentage	61
Available Page File	3456798720
Available Physical Memory	1414131712

Frame 	Module 	Signature 	Source
0 	xul.dll 	nsPrefBranch::GetIntPref 	modules/libpref/src/nsPrefBranch.cpp:181
1 	QIPCAP.dll 	QIPCAP.dll@0x1e840 	
2 	QIPCAP.dll 	QIPCAP.dll@0x1f726 	
3 	QIPCAP.dll 	QIPCAP.dll@0x214e4 	
4 	xul.dll 	nsObserverService::NotifyObservers 	xpcom/ds/nsObserverService.cpp:149
5 	xul.dll 	nsCOMPtr_base::~nsCOMPtr_base 	obj-firefox/dist/include/nsCOMPtr.h:408
6 	mozalloc.dll 	mozalloc.dll@0x109f 	
7 		@0xd 	
8 	mozglue.dll 	je_free 	memory/mozjemalloc/jemalloc.c:6589
9 	xul.dll 	nsLocalFile::Release 	xpcom/io/nsLocalFileWin.cpp:977
10 	xul.dll 	nsCOMPtr_base::~nsCOMPtr_base 	obj-firefox/dist/include/nsCOMPtr.h:408
11 	xul.dll 	NS_InitXPCOM2_P 	xpcom/build/nsXPComInit.cpp:447
12 	xul.dll 	ScopedXPCOMStartup::Initialize 	toolkit/xre/nsAppRunner.cpp:1181
13 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:3935
14 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:105
15 	firefox.exe 	__tmainCRTStartup 	crtexe.c:552
16 	ntdll.dll 	__RtlUserThreadStart 	
17 	crypt32.dll 	ErrLog_LogError 	
18 	kernel32.dll 	WerpReportFaultInternal 	
19 	kernel32.dll 	WerpReportFaultInternal 	

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nsPrefBranch%3A%3AGetIntPref%28char+const*%2C+int*%29

Comment 1

[Jorge Villalobos [:jorgev] (he/him)]

I've attempted to contact Websense about this.

Comment 2

[bhavana bajaj [:bajaj]]

CCing bsmedberg here to check if this could be related to https://bugzilla.mozilla.org/show_bug.cgi?id=828296 & recommendation regarding going ahead and blocklisting here.

Comment 3

[Benjamin Smedberg]

Yes, this is very likely related to bug 828296, and the extension needs to be recompiled to take that into account.

Comment 4

[Jorge Villalobos [:jorgev] (he/him)]

I don't see any extensions in the crash report. Maybe this is a DLL being injected into Firefox in some other way?

Comment 5

[Benjamin Smedberg]

Indeed. But it's still using XPCOM APIs after it manages to get itself injected.

Comment 6

[Tyler Downer [He/Him]]

This is starting to appear on SUMO. Any word from websense?

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

No response yet. I'm adding the only contact on Bugzilla that seems to still be valid.

Note that this would need to be an in-product DLL block if we chose to go that way.

Comment 8

[Alex Keybl [:akeybl]]

(In reply to Jorge Villalobos [:jorgev] from comment #7)
> No response yet. I'm adding the only contact on Bugzilla that seems to still
> be valid.
> 
> Note that this would need to be an in-product DLL block if we chose to go
> that way.

I'm also trying to give WebSense a call right now. I want to verify that they either plan to fix the issue in the short term for 7.6 users, or what the user experience will be like if we block the DLL.

(In reply to Tyler Downer [:Tyler] from comment #6)
> This is starting to appear on SUMO.

For reference: https://support.mozilla.org/en-US/questions/946986?esab=a&s=websense&r=1&as=s
And crash comments: https://crash-stats.mozilla.com/report/list?range_value=7&range_unit=days&date=2013-01-14&signature=nsPrefBranch%3A%3AGetIntPref%28char%20const*%2C%20int*%29&version=Firefox%3A18.0#comments

Comment 9

[Alex Keybl [:akeybl]]

No dice on the phone. Somebody will call me back, but the technical support people need to be contacted by customers. We need to grab some emails to perform outreach, so that their IT can contact Websense.

Is there any evidence that v7.6 of QIPCAP.dll is working for some users? If not, we should go ahead and blocklist.

Comment 10

[Alex Keybl [:akeybl]]

Marcia helped with the emails, since she's PT.

Comment 11

[Benjamin Smedberg]

Note that the block would have to be in-product, given that this is a DLL block.

Comment 12

[Benjamin Smedberg]

Attachment: Block qipcap.dll, rev. 1

Comment 13

[Robert Kaiser]

(In reply to Alex Keybl [:akeybl] from comment #9)
> Is there any evidence that v7.6 of QIPCAP.dll is working for some users?

I have no idea how I should find out about that.

Comment 14

[Scoobidiver (away)]

It happens with recent QIPCAP.dll versions, including the latest one:
  nsPrefBranch::GetIntPref(char const*, int*)|EXCEPTION_ACCESS_VIOLATION_WRITE (347 crashes)
    100% (347/347) vs.   0% (352/152000) QIPCAP.dll
          0% (0/347) vs.   0% (1/152000) 7.6.808.1
          0% (0/347) vs.   0% (1/152000) 7.6.811.1
          3% (9/347) vs.   0% (11/152000) 7.6.812.1
          3% (10/347) vs.   0% (11/152000) 7.6.813.1
         95% (328/347) vs.   0% (328/152000) 7.6.815.1

Comment 15

[(no longer active)]

Comment on attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1

Review of attachment 702323 [details] [diff] [review]:
-----------------------------------------------------------------

r=me

Comment 16

[(no longer active)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1fecb291afcc

Comment 17

[(no longer active)]

Comment on attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1

This patch is not risky and doesn't change strings or uuids.

Comment 18

[Benjamin Smedberg]

Comment on attachment 702323 [details] [diff] [review]
Block qipcap.dll, rev. 1

There's no reason to take this for the ESRs because the crash was caused by an interface change only in FF18.

Comment 19

[bhavana bajaj [:bajaj]]

Adding qawanted to help with verification that blocklist works as expected.

Comment 20

[Benjamin Smedberg]

I don't expect the crash to be reproduceable on nightly because we fixed the nsIPrefBranch IID.

Comment 21

[Scoobidiver (away)]

Should we land this patch only in Release and Beta, and not in Aurora and m-c where bug 828296 is fixed?

Comment 22

[Benjamin Smedberg]

The DLL is very likely not going to work in m-c/aurora, so I don't think there's harm in blocking it there also, and not blocking it would create a weird situation where it's loaded into a later version but not an earlier version.

Comment 23

[(no longer active)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/1f852f68169c
https://hg.mozilla.org/releases/mozilla-beta/rev/dc288063bac6
https://hg.mozilla.org/releases/mozilla-release/rev/6c5332ee34d4

Comment 24

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/1fecb291afcc

Comment 25

[juan becerra [:juanb]]

I tried to find Websense Endpoint in their product list, but after registering and getting a trial account I'm still waiting for their suite to finish downloading (2gb+).

It looks like it is a content filtering system, and most search results were about bypassing it through a proxy. I'll look into it tomorrow when it finishes downloading.

Comment 26

[juan becerra [:juanb]]

I installed the web security part of the test suite, but after installing there was no process loading the qipcap.dll (as show using the program ListDlls). Firefox 18.0 didn't load it, and as far as I can't tell, it wasn't even found in the installation I performed.

If anyone has any ideas, please let me know.

Comment 27

[netpq]

You don't need to fix because Websense did. FF 17& 18 dumped nsIPrefBranch2 interface and changed it's ID.

Comment 28

[Scoobidiver (away)]

It's still not fixed by the DLL blocklist:
  nsPrefBranch::GetIntPref(char const*, int*)|EXCEPTION_ACCESS_VIOLATION_WRITE (157 crashes)
    100% (157/157) vs.   0% (168/137016) QIPCAP.dll
          0% (0/157) vs.   0% (1/137016) 7.3.600.0
          1% (1/157) vs.   0% (6/137016) 7.6.812.1
          1% (1/157) vs.   0% (6/137016) 7.6.813.1
         99% (155/157) vs.   0% (155/137016) 7.6.815.1

(In reply to netpq from comment #27)
> You don't need to fix because Websense did. FF 17& 18 dumped nsIPrefBranch2
> interface and changed it's ID.
Where did you get this info?

Comment 29

[Robert Kaiser]

> (In reply to netpq from comment #27)
> > You don't need to fix because Websense did. FF 17& 18 dumped nsIPrefBranch2
> > interface and changed it's ID.
> Where did you get this info?

See bug 828296 - this is one of the more common causes of crashes with third-party binary software in Firefox 18. FYI, we suspect that the Norton Confidential crashes spiking in this release is connected to that as well, see the dependencies of that bug (this here is one of them, BTW).

Comment 30

[Scoobidiver (away)]

There are still crashes in 20.0a2 and 21.0a1 after the fix of bug 828296. See https://crash-stats.mozilla.com/report/list?version=Firefox:20.0a2&version=Firefox:21.0a1&signature=nsPrefBranch%3A%3AGetIntPref%28char+const*%2C+int*%29

Comment 31

[Benjamin Smedberg]

In that case it would be nice to figure out how the DLL is getting loaded and past our blocklist, but I don't plan on looking at this further.

Comment 32

[Alex Keybl [:akeybl]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #31)
> In that case it would be nice to figure out how the DLL is getting loaded
> and past our blocklist, but I don't plan on looking at this further.

Users have contacted Websense at this point and the crash remains quite low. I agree that we shouldn't devote more time to this issue specifically.

Comment 33

[bugzilla]

low crash depending of the organization, for us is a serious problem with my 700 users were most of them use Firefox and we must roll out websense to all the users, it crash on all Firefox versions > 17.x , tested in the beta versions as well.

Do we have to recomend stop using Firefox including dev teams that produce web portals for customers and let the customers know that Firefox is no supported ?

A bit ridiculous..

Comment 35

[Ioana (away)]

There are almost 30 crashes on Firefox 19 beta 5, which only got released a week ago:
https://crash-stats.mozilla.com/report/list?query_search=signature&query_type=contains&reason_type=contains&range_value=4&range_unit=weeks&hang_type=any&process_type=any&signature=nsPrefBranch%3A%3AGetIntPref%28char%20const%2A%2C%20int%2A%29

I've only seen 2 post-fix crashes on Firefox 20 though, and none on Firefox 21.

Comment 36

[Scoobidiver (away)]

(In reply to Ioana Budnar [QA] from comment #35)
> There are almost 30 crashes on Firefox 19 beta 5, which only got released a
> week ago:
The fix is not the DLL blocklist but the Websense Endpoint update so those who crash haven't updated it.

Comment 37

[bugzilla]

Just in case helps to someone,

I have been doing lots of testing latelly with Firefox 18.02 and different versions of websense enpoint client, the issue with the version 18.x seems to be resolved with websense endpoint version 1138 I just got today, unfortunatelly it is a preproduction version and the latest production version is 1130 to wich Firefox 18.x does not run therefore you need to request the version 1138 from websense support teams.

Comment 38

[Ioana (away)]

Considering the number of crashes in the last 4 weeks (5 on Fx 20, 0 on Fx 21 and 22), we can consider this bug fixed and verified.

Comment 40

[Swarnava Sengupta (:Swarnava)]

Firefox 21 beta crash id bp-92f342ba-bcfd-4ad0-9daf-c609a2130429

Comment 41

[Scoobidiver (away)]

(In reply to Swarnava Sengupta (:Swarnava) from comment #40)
> Firefox 21 beta crash id bp-92f342ba-bcfd-4ad0-9daf-c609a2130429
You are using a Websense Endpoint version that doesn't contain the fix (see the whiteboard).

Comment 42

[Swarnava Sengupta (:Swarnava)]

oh yes, i just told that user to update websense!

Comment 43

[Paul Pietromonaco]

Sorry for jumping in here, but I believe this is the correct location for this comment.  At work, we are running Websense Endpoint 1139.  The current version of Firefox - 21.xx - works fine, but the latest ESR version of Firefox - 17.0.6 - crashes upon startup.  If I disable Websense, ESR works normally.  There may not be any action required here - just mentioning that this is still affecting current ESR builds.

Comment 44

[Scoobidiver (away)]

(In reply to Paul Pietromonaco from comment #43)
> There may not be any action required here - just mentioning that this is still 
> affecting current ESR builds.
An extension with binary can't be compatible with 17.0 ESR and the current version if the binary has been changed after 17.0. Usually, ESR users don't need extensions.

Comment 45

[motospam]

(In reply to Paul Pietromonaco from comment #43)
> Sorry for jumping in here, but I believe this is the correct location for
> this comment.  At work, we are running Websense Endpoint 1139.  The current
> version of Firefox - 21.xx - works fine, but the latest ESR version of
> Firefox - 17.0.6 - crashes upon startup.  If I disable Websense, ESR works
> normally.  There may not be any action required here - just mentioning that
> this is still affecting current ESR builds.

I know this is off-topic, but:

Hi, Mr. Pietromonaco (Mr. Weasal-Bot, I presume?).   Alison Hinderliter (my ex-) from the Sockheads (Sifl & Olly) is looking for you on the Sockheads group on Facebook.    Hope you're doing well.  You can reach her at the Sockheads group here - https://www.facebook.com/groups/209138312440967/

Cheers,
Paul Caporino

Comment 46

[garrettmc]

Hello all, thank you for your efforts to collaborate. I want to provide information for direct contact regarding all product security issues in all Forcepoint products and services:

I am a founding member of the Forcepoint Global Product Security Incident Response Team (PSIRT).

(Forcepoint is the synthesis of Raytheon Cyber Products, Websense, Trusted Computer Solutions, Oakley Networks, Visual Analytics, Stonesoft, Sidewinder, and others)

If you ever need to report a product security vulnerability, contact [e-mail]. PGP Key: https://www.forcepoint.com/innovation/product-security/product-security-report-issue