1304434 - follow up - startup crash in nsPrefBranch::GetIntPref (Websense Endpoint)

Status: RESOLVED WORKSFORME

(External Software Affecting Firefox :: Other, defect, P3)

Description

[Marco Castelluccio [:marco]]

+++ This bug was initially created as a clone of Bug #1294650 +++

+++ This bug was initially created as a clone of Bug #1291738 +++

+++ This bug was initially created as a clone of Bug #828184 +++

We're still experiencing this crash with Firefox 49.0.

The affected version of the WebSense DLL is 7.6.818.1, it looks like they are
not versioning their DLL, so we can't know for sure which versions of WebSense
are affected.

I've analyzed the debug identifiers of the DLL on 48 and 49 to see if the DLLs
that are in the reports are the same between the two versions. The main ones are
the same, only the proportion is different.

  49 - 222 crash reports
    - 5BAD3E3071F74EBC9C642397D54EFB5C1: 164
    - C2E438B9BA7B47D7A4C7042B153620051: 33
    - 08610E0A65824CCCB34738CE21BCBD901: 3
    - 6AA150859B2D48FA8549AC80FB1A651B1: 15
    - 3E963A3FE7C64DB7B6CEBBC0570EA1A51: 1
    - 61354AEED16F43BCBE67B2E5C7FB1CEB1: 2
   
  48 - 679 crash reports
    - 5BAD3E3071F74EBC9C642397D54EFB5C1: 70
    - C2E438B9BA7B47D7A4C7042B153620051: 302
    - 08610E0A65824CCCB34738CE21BCBD901: 107
    - 6AA150859B2D48FA8549AC80FB1A651B1: 5
    - 644805ADA1E34F9A9EE3DC0A6B3697F41: 91
    - 024E18CBC9D84FEFA9799FE21BFFF0741: 64
    - 61D707D415524EF090CA8F6AC344C9AE1: 3
    - B2144E6EE70645FC97E620C92607389E1: 8

Comment 1

[[:philipp]]

maybe an interesting datapoint from today's firefox 49 addon correlation data for crashes: a proportion of crashing websense users seem to have the hotfix addon installed (and still got updated?!)

PREF_GettPref|EXCEPTION_ACCESS_VIOLATION_WRITE (291 crashes)
     17% (50/291) vs.   3% (356/10354) [e-mail]

Comment 2

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to [:philipp] from comment #1)
> maybe an interesting datapoint from today's firefox 49 addon correlation
> data for crashes: a proportion of crashing websense users seem to have the
> hotfix addon installed (and still got updated?!)
> 
> PREF_GettPref|EXCEPTION_ACCESS_VIOLATION_WRITE (291 crashes)
>      17% (50/291) vs.   3% (356/10354) [e-mail]

I tested updates when we released Firefox 49, with the following results:
1. NO Hotfix applied
   a. Firefox versions < 48.0 get updated to 47.0.1
   b. Firefox versions 48.x get updated to 49.0
2. Hotfix applied, and NO qipcap.dll detected
   a. Firefox versions < 48.0 get updated to 48.0.2 (due to watershed from bug 1284903), and then 48.0.2 updates to 49.0
   b. Firefox versions 48.x get updated directly to 49.0
   c. After update to 49.0, in all cases, app.update.url has NO more “websense”/”nowebsense” parameter, and the Hotfix NO longer appears as installed
   d. Tested manually on Win 7 x64, with: 46.0.1-de, 47.0-it, 48.0-en-US, 48.0.1-fr, 48.0.2-ro
3. Hotfix applied, and qipcap.dll detected in Windows/System32 folder
   a. Firefox versions < 48.0 get updated to 47.0.1 (after the update, app.update.url still contains the “websense” parameter, and the Hotfix is still installed)
   b. Firefox versions 48.x get updated to 49.0
   c. After update to 49.0, in all cases, app.update.url has NO more “websense”/”nowebsense” parameter, and the Hotfix NO longer appears as installed
   d. Tested manually on Win 7 x64, with: 46.0.1-de, 47.0-it, 48.0-en-US, 48.0.1-fr, 48.0.2-ro

From these results it would seem like the only way to get to Firefox 49 with Websense installed is by updating from 48.x, which doesn't make much sense since 48.x wouldn't start in this case.

Comment 3

[Jim Mathies [:jimm]]

(In reply to Marco Castelluccio [:marco] from comment #0)
> The affected version of the WebSense DLL is 7.6.818.1, it looks like they are
> not versioning their DLL, so we can't know for sure which versions of
> WebSense are affected.
> 
> I've analyzed the debug identifiers of the DLL on 48 and 49 to see if the
> DLLs that are in the reports are the same between the two versions. The main ones
> are the same, only the proportion is different.
> 
>   49 - 222 crash reports
>     - 5BAD3E3071F74EBC9C642397D54EFB5C1: 164
>     - C2E438B9BA7B47D7A4C7042B153620051: 33
>     - 08610E0A65824CCCB34738CE21BCBD901: 3
>     - 6AA150859B2D48FA8549AC80FB1A651B1: 15
>     - 3E963A3FE7C64DB7B6CEBBC0570EA1A51: 1
>     - 61354AEED16F43BCBE67B2E5C7FB1CEB1: 2
>    

According to the initial feedback we have they do version their dlls. Though I'm seeing the same thing you are, I did a quick pull of 200 random crash reports for 49.0 and found the same result - all QIPCAP.DLL/QIPCAP64.DLL versions in the reports were 7.6.818.1.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

Release Note Request (optional, but appreciated)
[Why is this notable]: Known issue that may affect 49 users who have older versions of Websense
[Suggested wording]: Firefox users who have older versions of Websense (before Websense 8.1) may hit a persistent startup crash. 
[Links (documentation, blog post, etc)]:   [SUMO post to come]

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Jim Mathies [:jimm] from comment #3)
> According to the initial feedback we have they do version their dlls. Though
> I'm seeing the same thing you are, I did a quick pull of 200 random crash
> reports for 49.0 and found the same result - all QIPCAP.DLL/QIPCAP64.DLL
> versions in the reports were 7.6.818.1.

In 48, it was the same.

Perhaps they are updating the version only on major releases?

If we had a way to match the debug IDs to the versions (I guess they could share
this info with us), we could see which versions are affected.

Comment 6

[Liz Henry (:lizzard) (relman/hg->git project)]

Rewording the relnote: 
"Older installations of Websense (before version 8.1) may cause a startup crash for Windows users"

Comment 8

[Liz Henry (:lizzard) (relman/hg->git project)]

We still have at least a few of these crashes on every current version. Marking them affected and it may make sense to track this across all versions.

Comment 9

[Marcia Knous [:marcia]]

Tracking 52+ as I have definitely seem some recent crashes in nightly that had the Websense dll.

Comment 10

[Gerry Chang [:gchang]]

Track 51+ as there are still some crashes in 51 aurora.

Comment 11

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Tracked though it remains to be seen what the extent of this issue is on 50, we will know once it goes live next week.

Comment 12

[Liz Henry (:lizzard) (relman/hg->git project)]

This signature is under 100 crashes per week on release, now. 
We still potentially could fix this for 51 (for January 24 release) so I'll leave it tracked.

Comment 13

[Liz Henry (:lizzard) (relman/hg->git project)]

266 crashes in the last week with 50.1. 
53 is still affected. Marking fix-optional for 51 as it seems unlikely we will have a sure fix by the 51 release Jan. 23.

Comment 14

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

One update here: based on very recent testing [1] done with various Websense Endpoint versions on Win 10 x64, we get this crash on startup of Firefox when Endpoint version 8.2 (8.2.2324) is installed. Fx 50.1.0 crashes on startup, while Fx 47.0.2 doesn't. Also, there were no crashes encountered with more recent versions of the Endpoint like: 8.2.5 (8.2.5.2424) and 8.3 (8.3.2509).

[1] - https://docs.google.com/spreadsheets/d/18n8NEimajNpnGRHdQsrrnyJ-Ufgvmpz-AXCUpVaFRMk/edit#gid=0

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Liz, do you still want to add this to the release notes?
(btw don't hesitate to do it directly)

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Julien is telling me that it is already in 49 release notes, updating the flag accordingly.

Comment 17

[Julien Cristau [:jcristau]]

Too late for firefox 52, mass-wontfix.

Comment 18

[Jim Mathies [:jimm]]

websense converted to a webextension in newer revs. older revs are a good blicklist candidate.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

We're still shipping the dummy qipcap64.dll to workaround this ancient issue. Can we use this bug to remove that hack and maybe just add it to the DLL blocklist now that we have better mechanisms for doing so than we had at the time?

Comment 20

[Jim Mathies [:jimm]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #19)

We're still shipping the dummy qipcap64.dll to workaround this ancient issue. Can we use this bug to remove that hack and maybe just add it to the DLL blocklist now that we have better mechanisms for doing so than we had at the time?

No assurances new methods will fix this. In some cases (kernel drivers) we can't block. Since the old rev of their software is long since gone we don't know if blocking this dll will fix the problem. We can remove this in like 5 years or so, but I wouldn't drop it now. That software is still out there.

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.