1291738 - startup crash in nsPrefBranch::GetIntPref (Websense Endpoint)

Status: RESOLVED FIXED

(External Software Affecting Firefox :: Other, defect)

Description

[Sylvestre Ledru [:Sylvestre]]

+++ This bug was initially created as a clone of Bug #828184 +++

It's #5 top browser crasher in the first hours of 18.0.
It's a startup crash but with not so many duplicates.

The last time, it was caused by Websense Endpoint (https://www.websense.com/content/Home.aspx).

Comment 1

[Sylvestre Ledru [:Sylvestre]]

Tracking as it is a startup crash and a top crash

Comment 2

[Sylvestre Ledru [:Sylvestre]]

I sent an email to Websense.

Benjamin, when it happened in bug #828184, we blocked them, should we do the same here?

Comment 3

[Marcia Knous [:marcia]]

The first few crash reports I clicked on all had QIPCAP.dll 7.6.818.1 in the module section, which appears to be the Websense DLL. Need to get the full correlation report to see the full scope.

Will also check for Mac crashes as in the past they have come up with a signature containing qipcap.dll.

Comment 5

[Benjamin Smedberg]

Need a product manager (pdol?) to make a decision here: we typically try to contact the 3rd-party software author. Determine whether to block paticular versions or all versions. It would be useful to understand what they are doing, because if it's binary XPCOM that needs to die quickly.

Comment 6

[Loic]

A user on SUMO reported today crash on startup with the same crash signature:
https://support.mozilla.org/en-US/questions/1133501

Comment 7

[Sylvestre Ledru [:Sylvestre]]

Sander, I tried to contact them on [e-mail]  but I received an error. Do you have access to their platform? I don't see where I can contact them.

If they only support ESR as it has been said in https://bugzilla.mozilla.org/show_bug.cgi?id=1181091#c57, we should maybe blocklist them from every firefox release

Comment 8

[Sylvestre Ledru [:Sylvestre]]

Attachment: Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Review commit: https://reviewboard.mozilla.org/r/69542/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69542/

Comment 9

[Benjamin Smedberg]

Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

https://reviewboard.mozilla.org/r/69542/#review66728

Comment 10

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69542/diff/1-2/

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a08c6285367
Block the version of websense crashing 48 & 49 r=bsmedberg

Comment 12

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Approval Request Comment
[Feature/regressing bug #]: Outside change
[User impact if declined]: Startup crash
[Describe test coverage new/current, TreeHerder]: I don't have a windows to test, the patch is trivial
[Risks and why]: We are dealing with startup crashes, we cannot make it worst...
[String/UUID change made/needed]: None

Comment 13

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

DLL blocklist for a startup crash, Release48+, Beta49+, Aurora50+

Comment 14

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/8a08c6285367

Comment 15

[u194118]

Still crashing for me.
https://crash-stats.mozilla.com/report/index/bp-095279ae-a7b5-4f22-9ead-c1aef2160808

Should I do something to benefit from the dll blocking ?

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8a2e696c8bd1

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-beta/rev/0c78321d9c7d

Comment 18

[Sylvestre Ledru [:Sylvestre]]

(In reply to jcleval from comment #15)
> Should I do something to benefit from the dll blocking ?
We, Mozilla, maintain 4 versions of Firefox in parallel (nightly, aurora/dev edition, beta and release). Here, we took the fix only for the first one. Carsten is currently applying the change to other branches. 
Once it is done, we will start a new build including the change. This week or the next.

You can check if the crash still occurs for you on the latest nightly:
http://nightly.mozilla.org/

Comment 19

[Sylvestre Ledru [:Sylvestre]]

Clearing the need info to Sander, he replied to me by email.

Comment 20

[Carsten Book [:Tomcat]]

 https://hg.mozilla.org/releases/mozilla-release/rev/c561088c8862

Comment 21

[Peter Dolanjski [:pdol]]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #5)
> Need a product manager (pdol?) to make a decision here: we typically try to
> contact the 3rd-party software author. Determine whether to block paticular
> versions or all versions. It would be useful to understand what they are
> doing, because if it's binary XPCOM that needs to die quickly.

Sorry for the delay (was on PTO).  Looks like the decision was made, which seems to be the right one from my perspective.

Comment 22

[[:philipp]]

the patch has gone into 49.0b2, but it doesn't seem to be effective as those crashes continue and reports like https://crash-stats.mozilla.com/report/index/5cd1d208-a134-426f-bb0d-486e22160810 show that the qipcap.dll module is still hooking into the process.

Comment 23

[Sander Goudswaard]

It may help to ship a qipcap.dll in the same directory - a dll that does nothing but overrules the normal one. Not sure if this is (legally) possible but I tested this and seems to work :)

Comment 24

[Marco Castelluccio [:marco]]

Could the issue be that there are two entries in the DLL blocklist with the same file name?

Comment 25

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #24)
> Could the issue be that there are two entries in the DLL blocklist with the
> same file name?

I think the loop in patched_LdrLoadDll uses the first matching entry from sWindowsDllBlocklist:

  info = &sWindowsDllBlocklist[0];
  while (info->name) {
    if (strcmp(info->name, dllName) == 0)
      break;

    info++;
  }

So in this case it will use {"qipcap.dll", MAKE_VERSION(7, 6, 815, 1)} instead of { "qipcap.dll", MAKE_VERSION(7, 6, 818, 1) }, or am I missing something?

Comment 26

[(No longer employed by Mozilla) Aaron Klotz]

The version field is a maximum, that is, we block anything that is less-than or equal to that version. You should have a single line that just blocks 7.6.818.1 and it will take care of older versions as well.

Comment 27

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8778181 [details]
Bug 1291738 - Block the version of websense crashing 48 & 49. Followup, we already had this dll

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69542/diff/2-3/

Comment 28

[Sylvestre Ledru [:Sylvestre]]

Closing (maybe fixed is not the best) as it has been managed in bug 1294650

Comment 29

[Gerry Chang [:gchang]]

Added in the 48.0.1 release notes: "Fix a startup crash issue caused by Websense".

Comment 30

[Mihai Boldan, Desktop QA [:mboldan]]

I was not able to reproduce this issue on Firefox 47 or on Firefox 48, under Windows 7x86 and under Windows 8x86, using the STR from Comment 0.
Is there a specific platform, build or other STR in order to reproduce this crash?

Note that the issue was tested under Windows 7x86 on two different PC's.

Comment 31

[garrettmc]

Hello all, thank you for your efforts to collaborate. I want to provide information for direct contact regarding all product security issues in all Forcepoint products and services:

I am a founding member of the Forcepoint Global Product Security Incident Response Team (PSIRT).

(Forcepoint is the synthesis of Raytheon Cyber Products, Websense, Trusted Computer Solutions, Oakley Networks, Visual Analytics, Stonesoft, Sidewinder, and others)

If you ever need to report a product security vulnerability, contact [e-mail]. PGP Key: https://www.forcepoint.com/innovation/product-security/product-security-report-issue