startup crash in nsPrefBranch::GetIntPref (Websense Endpoint)

RESOLVED FIXED

Status

defect
--
critical
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: sylvestre, Assigned: sylvestre)

Tracking

(Depends on 1 bug, Blocks 1 bug, {crash, topcrash})

unspecified
x86
Windows 7
Dependency tree / graph
Bug Flags:
qe-verify +

Firefox Tracking Flags

(firefox48blocking fixed, firefox49 fixed, relnote-firefox 48+, firefox50 fixed, firefox51 fixed)

Details

(Whiteboard: [startupcrash], crash signature)

Attachments

(1 obsolete attachment)

+++ This bug was initially created as a clone of Bug #828184 +++

It's #5 top browser crasher in the first hours of 18.0.
It's a startup crash but with not so many duplicates.

The last time, it was caused by Websense Endpoint (https://www.websense.com/content/Home.aspx).
Tracking as it is a startup crash and a top crash
Crash Signature: [@ nsPrefBranch::GetIntPref(char const*, int*)] [@ PREF_GetIntPref ] → [@ PREF_GetIntPref ]
I sent an email to Websense.

Benjamin, when it happened in bug #828184, we blocked them, should we do the same here?
Flags: needinfo?(benjamin)
The first few crash reports I clicked on all had QIPCAP.dll 7.6.818.1 in the module section, which appears to be the Websense DLL. Need to get the full correlation report to see the full scope.

Will also check for Mac crashes as in the past they have come up with a signature containing qipcap.dll.
Component: General → Other
Product: Core → External Software Affecting Firefox
Version: 18 Branch → unspecified
Duplicate of this bug: 1291736
Need a product manager (pdol?) to make a decision here: we typically try to contact the 3rd-party software author. Determine whether to block paticular versions or all versions. It would be useful to understand what they are doing, because if it's binary XPCOM that needs to die quickly.
Flags: needinfo?(benjamin) → needinfo?(pdolanjski)
A user on SUMO reported today crash on startup with the same crash signature:
https://support.mozilla.org/en-US/questions/1133501
Sander, I tried to contact them on support@websense.com  but I received an error. Do you have access to their platform? I don't see where I can contact them.

If they only support ESR as it has been said in https://bugzilla.mozilla.org/show_bug.cgi?id=1181091#c57, we should maybe blocklist them from every firefox release
Flags: needinfo?(sander+bugzilla)
Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

https://reviewboard.mozilla.org/r/69542/#review66728
Attachment #8778181 - Flags: review?(benjamin) → review+
Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69542/diff/1-2/
Pushed by sledru@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8a08c6285367
Block the version of websense crashing 48 & 49 r=bsmedberg
Assignee: nobody → sledru
Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Approval Request Comment
[Feature/regressing bug #]: Outside change
[User impact if declined]: Startup crash
[Describe test coverage new/current, TreeHerder]: I don't have a windows to test, the patch is trivial
[Risks and why]: We are dealing with startup crashes, we cannot make it worst...
[String/UUID change made/needed]: None
Attachment #8778181 - Flags: approval-mozilla-release?
Attachment #8778181 - Flags: approval-mozilla-beta?
Attachment #8778181 - Flags: approval-mozilla-aurora?
Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

DLL blocklist for a startup crash, Release48+, Beta49+, Aurora50+
Attachment #8778181 - Flags: approval-mozilla-release?
Attachment #8778181 - Flags: approval-mozilla-release+
Attachment #8778181 - Flags: approval-mozilla-beta?
Attachment #8778181 - Flags: approval-mozilla-beta+
Attachment #8778181 - Flags: approval-mozilla-aurora?
Attachment #8778181 - Flags: approval-mozilla-aurora+
https://hg.mozilla.org/mozilla-central/rev/8a08c6285367
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Still crashing for me.
https://crash-stats.mozilla.com/report/index/bp-095279ae-a7b5-4f22-9ead-c1aef2160808

Should I do something to benefit from the dll blocking ?
(In reply to jcleval from comment #15)
> Should I do something to benefit from the dll blocking ?
We, Mozilla, maintain 4 versions of Firefox in parallel (nightly, aurora/dev edition, beta and release). Here, we took the fix only for the first one. Carsten is currently applying the change to other branches. 
Once it is done, we will start a new build including the change. This week or the next.

You can check if the crash still occurs for you on the latest nightly:
http://nightly.mozilla.org/
Clearing the need info to Sander, he replied to me by email.
Flags: needinfo?(sander+bugzilla)
(In reply to Benjamin Smedberg [:bsmedberg] from comment #5)
> Need a product manager (pdol?) to make a decision here: we typically try to
> contact the 3rd-party software author. Determine whether to block paticular
> versions or all versions. It would be useful to understand what they are
> doing, because if it's binary XPCOM that needs to die quickly.

Sorry for the delay (was on PTO).  Looks like the decision was made, which seems to be the right one from my perspective.
Flags: needinfo?(pdolanjski)
Flags: qe-verify+
the patch has gone into 49.0b2, but it doesn't seem to be effective as those crashes continue and reports like https://crash-stats.mozilla.com/report/index/5cd1d208-a134-426f-bb0d-486e22160810 show that the qipcap.dll module is still hooking into the process.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
It may help to ship a qipcap.dll in the same directory - a dll that does nothing but overrules the normal one. Not sure if this is (legally) possible but I tested this and seems to work :)
Could the issue be that there are two entries in the DLL blocklist with the same file name?
(In reply to Marco Castelluccio [:marco] from comment #24)
> Could the issue be that there are two entries in the DLL blocklist with the
> same file name?

I think the loop in patched_LdrLoadDll uses the first matching entry from sWindowsDllBlocklist:

  info = &sWindowsDllBlocklist[0];
  while (info->name) {
    if (strcmp(info->name, dllName) == 0)
      break;

    info++;
  }

So in this case it will use {"qipcap.dll", MAKE_VERSION(7, 6, 815, 1)} instead of { "qipcap.dll", MAKE_VERSION(7, 6, 818, 1) }, or am I missing something?
Flags: needinfo?(benjamin)
Flags: needinfo?(aklotz)
The version field is a maximum, that is, we block anything that is less-than or equal to that version. You should have a single line that just blocks 7.6.818.1 and it will take care of older versions as well.
Flags: needinfo?(benjamin)
Flags: needinfo?(aklotz)
Attachment #8778181 - Attachment is obsolete: true
Closing (maybe fixed is not the best) as it has been managed in bug 1294650
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Added in the 48.0.1 release notes: "Fix a startup crash issue caused by Websense".
I was not able to reproduce this issue on Firefox 47 or on Firefox 48, under Windows 7x86 and under Windows 8x86, using the STR from Comment 0.
Is there a specific platform, build or other STR in order to reproduce this crash?

Note that the issue was tested under Windows 7x86 on two different PC's.
Depends on: 1304434
Hello all, thank you for your efforts to collaborate. I want to provide information for direct contact regarding all product security issues in all Forcepoint products and services:

I am a founding member of the Forcepoint Global Product Security Incident Response Team (PSIRT).

(Forcepoint is the synthesis of Raytheon Cyber Products, Websense, Trusted Computer Solutions, Oakley Networks, Visual Analytics, Stonesoft, Sidewinder, and others)

If you ever need to report a product security vulnerability, contact PSIRT@forcepoint.com. PGP Key: https://www.forcepoint.com/innovation/product-security/product-security-report-issue
You need to log in before you can comment on or make changes to this bug.