1298404 - Deploy a hotfix to Firefox 47 (and lower) users to prevent them from updating if they have WebSense installed

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect)

Description

[Benjamin Smedberg]

Firefox 48 crashes at startup when Websense is installed (see bug 1294650). Right now we don't know how to prevent this crash, but we want to update the users who don't experience this crash.

Brad has suggested that we should deploy a system addon to users on FF47 and lower which detects websense, and if websense is not installed triggers an update.

I have made a stab at a detailed user story for this, but I'd like confirmation both from Brad/Jim on the proper websense detection, and from bhearsum/releng on the specifics of changes to the AUS URL.

Comment 1

[Brad Lassey [:blassey] (use needinfo?)]

Looking at the crash reports, this should be win64 as well as win32.

Comment 2

[Benjamin Smedberg]

Should we look in a different location than windir/system32/qipcap.dll?

Comment 3

[David Durst [:ddurst]]

Confused: user story doesn't say that we're triggering an update. Is the goal just to adjust the AUS URL and leave the update/not-update to the user?

Comment 4

[Jim Mathies [:jimm]]

Rather than checking for the dll in system32, a check using GetModuleHandle("QIPCAP.DLL" / "QIPCAP64.DLL") from within the firefox process should tell us if the software is loaded into our process space.  Is this possible or does the code we'll add run independent of firefox.exe?

Comment 5

[Benjamin Smedberg]

The code could have access to ctypes, but I don't think that check is as reliable as we'd like: we don't know what triggers qipcap in the first place, so it could be dependent on external factors. (Also that's more risk because the ctypes code to do that is hard)

Comment 6

[Jim Mathies [:jimm]]

Alright, according to crash dumps, we're searching for QIPCAP.DLL (32-bit firefox) or QIPCAP64.DLL (64-bit firefox), both located in windows System32 directory. We don't really need to worry about 64-bit if we don't want too, current crash rates indicate very few 64-bit users are impacted by this.

Comment 7

[[e-mail] (:bhearsum)]

(Adding Osmose, who you'll likely need when it comes to shipping the system addon.)

We don't support routing updates by arbitrary query parameters in Balrog. How about altering %OS_VERSION% or %DISTRIBUTION% instead? That should make it quick and easy to add a block for WebSense folks. %OS_VERSION% is the typical place we put odds and ends (like eg: gtk versions).

Comment 8

[:Felipe Gomes (needinfo for replies!)]

Important note: to update the app.update.url pref we'll need to update it through the default pref branch, because that's what's the updater uses.  This means that it needs to be done on every startup, and not just once to save the pref. The good thing is that with this, it doesn't incur any of the complicated pref clean-up whenever we don't need this anymore.


I'd like to propose to go with a Hotfix instead of a System Addon because the process will be much simpler imo. With just one signed xpi we can target users starting from Firefox 39 (bug 1151537), and there won't be the need to repackage and ensure that all shipped system add-ons are correctly included with their right versions.

Comment 9

[Benjamin Smedberg]

it's not easy to alter the existing OS_VERSION or DISTRIBUTION from a hotfix/system addon because those are substituted in builtin code.

Comment 10

[Molly Howell (she/her) (mostly inactive)]

We could remove the OS_VERSION or DISTRIBUTION tag from the URL that we override with, and fill in that data ourselves.

Also, we have a separate app.update.url.override pref, which isn't taken from the defaults and overrides the normal URL if it's set. But avoiding the cleanup that Felipe mentioned is probably worth doing it the harder way.

Comment 11

[Robert Strong (they/them - no direct email)]

The info could be added to the distribution.id default branch preference which replaces %DISTRIBUTION% in the the app.update.url preference. It would need to be done on every startup as Felipe noted.

Comment 12

[David Durst [:ddurst]]

Repeating question per Liz request, this time to Sylvestre (others may weigh in as well): is the goal just to adjust the AUS URL and leave the update/not-update decision to the user?

Comment 13

[[e-mail] (:bhearsum)]

(In reply to David Durst [:ddurst] from comment #12)
> Repeating question per Liz request, this time to Sylvestre (others may weigh
> in as well): is the goal just to adjust the AUS URL and leave the
> update/not-update decision to the user?

My understanding is that we are simply going to block these users from updating. I'm not sure we have a way to present an update that requires user opt-in anymore. We used to be able to do it with update_type = "minor", but that may not be supported by Firefox anymore. I don't think we need to wait to have answer on this before we block these users from updating, though, we can continue thinking about a solution for them later.

Comment 14

[Osmose [:osmose, :mkelly]]

(Adding rdalal in case we want to ship a system add-on while I'm out on PTO next week.)

Comment 15

[Robert Strong (they/them - no direct email)]

(In reply to Ben Hearsum (:bhearsum) from comment #13)
> (In reply to David Durst [:ddurst] from comment #12)
> > Repeating question per Liz request, this time to Sylvestre (others may weigh
> > in as well): is the goal just to adjust the AUS URL and leave the
> > update/not-update decision to the user?
> 
> My understanding is that we are simply going to block these users from
> updating. I'm not sure we have a way to present an update that requires user
> opt-in anymore. We used to be able to do it with update_type = "minor", but
> that may not be supported by Firefox anymore. I don't think we need to wait
> to have answer on this before we block these users from updating, though, we
> can continue thinking about a solution for them later.
You can still require that the user opt-in to getting the update just as we used to require opt-in with the old update url format when update type = "major" and showed the old billboard by adding the showPrompt = "true" to the update.xml.
https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/nsIUpdateService.idl#158

I don't think this would be valuable since this doesn't include any custom messaging.

Comment 16

[Bob Owen (:bobowen)]

(In reply to Jim Mathies [:jimm] from comment #6)
> Alright, according to crash dumps, we're searching for QIPCAP.DLL (32-bit
> firefox) or QIPCAP64.DLL (64-bit firefox), both located in windows System32
> directory. We don't really need to worry about 64-bit if we don't want too,
> current crash rates indicate very few 64-bit users are impacted by this.

If it is 32-bit Firefox on 64-bit Windows QIPCAP.dll will be in SysWOW64 not System32:
https://crash-stats.mozilla.com/report/index/8d1afa08-018c-4eae-8e91-dbea22160826

But maybe QIPCAP64.DLL will also be in System32 in this case.

Comment 17

[:Felipe Gomes (needinfo for replies!)]

From https://msdn.microsoft.com/en-us/library/windows/desktop/aa384187%28v=vs.85%29.aspx:
In order for 32-bit Firefox on 64-bit Windows to check System32 instead of SysWOW64 we can use sysnative. To be thorough, do people think if we should check all four following paths?

C:\Windows\System32\qipcap.dll
C:\Windows\System32\qipcap64.dll

C:\Windows\sysnative\qipcap.dll
C:\Windows\sysnative\qipcap64.dll

Or does anyone have a definitive answer of which of them can be skipped?

Comment 18

[:Felipe Gomes (needinfo for replies!)]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #9)
> it's not easy to alter the existing OS_VERSION or DISTRIBUTION from a
> hotfix/system addon because those are substituted in builtin code.

We could just replace "%OS_VERSION%" by "%OS_VERSION%-websense" in app.update.url, for example. That's probably better than changing the distribution.id pref because something else also updates this pref dynamically and they would be racing.

Comment 19

[Robert Strong (they/them - no direct email)]

(In reply to :Felipe Gomes (needinfo me!) from comment #18)
> (In reply to Benjamin Smedberg [:bsmedberg] from comment #9)
> > it's not easy to alter the existing OS_VERSION or DISTRIBUTION from a
> > hotfix/system addon because those are substituted in builtin code.
> 
> We could just replace "%OS_VERSION%" by "%OS_VERSION%-websense" in
> app.update.url, for example. That's probably better than changing the
> distribution.id pref because something else also updates this pref
> dynamically and they would be racing.

The distribution code updates that pref. I'm good with just appending it to %OS_VERSION%.

Comment 20

[[e-mail] (:bhearsum)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #19)
> (In reply to :Felipe Gomes (needinfo me!) from comment #18)
> > (In reply to Benjamin Smedberg [:bsmedberg] from comment #9)
> > > it's not easy to alter the existing OS_VERSION or DISTRIBUTION from a
> > > hotfix/system addon because those are substituted in builtin code.
> > 
> > We could just replace "%OS_VERSION%" by "%OS_VERSION%-websense" in
> > app.update.url, for example. That's probably better than changing the
> > distribution.id pref because something else also updates this pref
> > dynamically and they would be racing.
> 
> The distribution code updates that pref. I'm good with just appending it to
> %OS_VERSION%.

We'll still have to use app.update.url.override though, right? Which has sometimes downsides according to comment #10?

Comment 21

[Robert Strong (they/them - no direct email)]

No, the add-on can update the default pref on startup per comment #8

Comment 22

[Jim Mathies [:jimm]]

(In reply to Bob Owen (:bobowen) (less responsive until 26th) from comment #16)
> (In reply to Jim Mathies [:jimm] from comment #6)
> > Alright, according to crash dumps, we're searching for QIPCAP.DLL (32-bit
> > firefox) or QIPCAP64.DLL (64-bit firefox), both located in windows System32
> > directory. We don't really need to worry about 64-bit if we don't want too,
> > current crash rates indicate very few 64-bit users are impacted by this.
> 
> If it is 32-bit Firefox on 64-bit Windows QIPCAP.dll will be in SysWOW64 not
> System32:
> https://crash-stats.mozilla.com/report/index/8d1afa08-018c-4eae-8e91-
> dbea22160826
> 
> But maybe QIPCAP64.DLL will also be in System32 in this case.

Good catch, we should check 

(Windows Dir)\SysWOW64\

as well. Alternatively we could make a LoadLibrary call and let the system handle the search, but I suppose that has the same draw backs GetModuleHandle has.

Comment 23

[:Felipe Gomes (needinfo for replies!)]

Attachment: First version

Here's a first version of a hotfix to do what's been talked about here. It checks the list of files from comment 17, and if any is present, it replaces "%OS_VERSION%" with "%OS_VERSION%-websense"

Comment 24

[Robert Strong (they/them - no direct email)]

Comment on attachment 8785412 [details] [diff] [review]
First version

Looks good!

Comment 25

[Jim Mathies [:jimm]]

(In reply to :Felipe Gomes (needinfo me!) from comment #17)
> From
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa384187%28v=vs.
> 85%29.aspx:
> In order for 32-bit Firefox on 64-bit Windows to check System32 instead of
> SysWOW64 we can use sysnative. To be thorough, do people think if we should
> check all four following paths?
> 
> C:\Windows\System32\qipcap.dll
> C:\Windows\System32\qipcap64.dll
> 
> C:\Windows\sysnative\qipcap.dll
> C:\Windows\sysnative\qipcap64.dll
> 
> Or does anyone have a definitive answer of which of them can be skipped?

This looks right, according to that article, 32-bit firefox will automatically be redirected to the SysWOW64 directory. So you have SysWOW64 and System32 covered for 32-bit firefox, and 64-bit firefox will end up checking System32 twice.

Comment 26

[:Felipe Gomes (needinfo for replies!)]

Attachment: Unsigned xpi

I fixed a typo locally (nsIlocalFile -> nsILocalFile) and packaged the xpi

Comment 27

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8785413 [details]
Unsigned xpi

Could someone sign this xpi and stage it on addons.allizom.org (according to bug 1151537 comment 69)?

Comment 28

[:Felipe Gomes (needinfo for replies!)]

Of note: I marked this hotfix compatible up to 48.*.  I think we should ship it to 48 too, so that the release of 49 is not blocked by this in case the crash is not yet fully solved by the time that 49 is to be released.

Comment 29

[[e-mail] (:bhearsum)]

Something I just thought of: how do we know when everyone has the hotfix? AFAICT, we can't deploy 48.0 until they do, because WebSense users will still look the same as non-WebSense users. Felipe says hotfixes are taken very quickly, so maybe this isn't a concern?

One thing we might be able to do to mitigate this is add "nowebsense" to the OS_VERSION for non-WebSense users, then we'd be able to do a positive and negative match in Balrog.

Comment 30

[:Felipe Gomes (needinfo for replies!)]

Attachment: Version 2

Changed the replacement string to be "(websense)", and now we're also adding "(nowebsense)" in order to make it safer to activate updates (so that we can know we are only updating users who have received the hotfix).

Comment 31

[:Felipe Gomes (needinfo for replies!)]

Attachment: Unsigned xpi, v2

Comment 32

[Benjamin Smedberg]

bhearsum, the point was to only update users who report "nowebsense" in their URL. Everyone else should get 47.0.x

Comment 33

[[e-mail] (:bhearsum)]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #32)
> bhearsum, the point was to only update users who report "nowebsense" in
> their URL. Everyone else should get 47.0.x

Okay, that wasn't clear until the last couple of comments - the original addon only changed OS_VERSION for WebSense users.

Comment 34

[Matthew N. [:MattN]]

Comment on attachment 8785427 [details] [diff] [review]
Version 2

Review of attachment 8785427 [details] [diff] [review]:
-----------------------------------------------------------------

::: v20160826.01/bootstrap.js
@@ +45,5 @@
> +    ];
> +
> +    // If any of the above files exist, update the app.update.url pref
> +    // to include a special mark, so that we can avoid shipping an update
> +    // to these installations until a related crash is solved.

This comment is slightly misleading now as we're modifying the app.update.url pref regardless of whether websense is found or not.

Is Balrog going to be able to offer updates to Windows users with nowebsense and non-Windows users without any URL change? We're not modifying the app.update.url for non-Windows users.

::: v20160826.01/install.rdf
@@ +12,3 @@
>  
> +    <em:name>Firefox Hotfix</em:name>
> +    <em:description>Firefox Hotfix</em:description>

I think it would be useful to have a real description here since this hotfix will remain installed for Windows users. This will help reduce the chances of users getting freaked out by this.

On the same note I think we should re-add the icons to the add-on so we don't have a generic add-on logo in the AOM. See past hotfixes where we bundle the two icons: http://hg.mozilla.org/releases/firefox-hotfixes/file/tip/v20121019.01

Comment 35

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/firefox-hotfixes/rev/c049655251a9

Comment 36

[Sylvestre Ledru [:Sylvestre]]

This is a great idea, thanks for proposing it as workaround.
Just to be clear, this must NOT be pushed today (Friday). We need a sign off from QE.

Comment 37

[:Felipe Gomes (needinfo for replies!)]

Attachment: Unsgined xpi with icons, ready for signing

Comment 38

[Sylvestre Ledru [:Sylvestre]]

Guys, could you verify this tomorrow? I would like us to go live Monday. Thanks

Comment 39

[Kris Maglione [:kmag]]

The add-on has been signed and staged to addons.allizom.org.

Please note that while I can still deploy hotfixes in an emergency, I don't really work in this area of AMO anymore. In the future, it would be better to ping Jorge and Andreas Wagner.

Comment 40

[:Felipe Gomes (needinfo for replies!)]

Here are the instructions on how to test this Hotfix from the staging server:

https://wiki.mozilla.org/Add-ons/Hotfix#Testing_the_hotfix_on_the_staging_server  (Also take a look at the "Triggering a background update check" session so you don't have to sit waiting for the updater timer to fire)

In one of the steps, it says "Check that the hotfix performed its action correctly". Here are the important points:

- Once the hotfix is installed, it will check if WebSense is installed in the machine, and update the "app.update.url" pref accordingly. Test scenarios with both WebSense installed or not to verify. See the User Story above with details.

- It should be verified that the change remains in effect after a restart.

- Please verify with Firefox 47, if possible use both 32-bit and 64-bit Firefox and 32-bit and 64-bit Windows.

- Also do some sanity checks with Firefox 39 to see if it works (no need to repeat all tests).

- For Mac/Linux, the hotfix will quickly install and uninstall itself and shouldn't touch the app.update.url pref.

- Note: on Windows, this hotfix won't uninstall itself (unlike most other hotfixes in the past)

Comment 41

[Andrei Vaida [:avaida]]

Hey Felipe, we're having trouble installing the hotfix on 47 and 39.

Here's the flow we've used:
  1. we created a dummy qipcap.dll/qipcap64.dll file and placed it in
  ...\System32\, ...\sysnative\ and ...\SysWOW64\ (as needed)

  2. we created a new profile and started it with the latest release first, to
     disable automatic updates (via about:preferences#advanced → Update → Check
     for updates, but let me choose whether to install them)

  3. we started 47/39 with that profile and:
    - set devtools.chrome.enabled to true
    - confirmed that extensions.hotfix.id is set to [e-mail]
    - set extensions.logging.enabled to true
    - set extensions.update.interval to 10
    - set extensions.update.background.url to https://versioncheck.allizom.org/
      update/VersionCheck.php?reqVersion=%REQ_VERSION%&id=%ITEM_ID%&version=%IT
      EM_VERSION%&maxAppVersion=%ITEM_MAXAPPVERSION%&status=%ITEM_STATUS%&appID
      =%APP_ID%&appVersion=%APP_VERSION%&appOS=%APP_OS%&appABI=%APP_ABI%&locale
      =%APP_LOCALE%&currentAppVersion=%CURRENT_APP_VERSION%&updateType=%UPDATE_
      TYPE%&compatMode=%COMPATIBILITY_MODE%
    - restarted the browser
    - ran the following snippet in the Browser Console:
        Components.utils.import("resource://gre/modules/AddonManager.jsm");
        AddonManagerPrivate.backgroundUpdateCheck();
    - checked about:addons and about:support for any signs of the hotfix: none
    found

We followed these steps on Windows 7 x64, Windows XP SP3 x86, Windows 8.1 x86
and Windows 10 x64 -- all of them with the same result.

Please note that:
- the hotfix does not install if update settings are set to either "Never..." or
"let me choose..."
- with default update settings the hotfix is installed (shows in Add-on
  Manager) but the app.update.url pref is not changed
- because of the above we cannot test versions <43.0.1 because they get updated
  to 43.0.1 before we can test

Also, perhaps a few logs from 39.0.3 (Windows XP SP3 x86) might help:
  http://pastebin.com/Fi6JrQrr

Comment 42

[:Felipe Gomes (needinfo for replies!)]

It looks like the problem is this error from the logs:

Signature Verification Error: the signature on this .jar archive is invalid because the certificate used to sign this file has an unrecognized issuer.
addons.manager  WARN    The hotfix add-on was not signed by the expected certificate and so will not be installed.: [Exception... "A required certificate was not present." ...


So it's not the right cert? Jorge, Andreas, do you know how to sign using the right cert for the hotfix?  It has to be one of these two: http://searchfox.org/mozilla-central/rev/a2812fa126be538f73efed589e78d6973f23df2f/browser/app/profile/firefox.js#57

Comment 43

[Andreas Wagner [:TheOne] [use NI]]

Looking at the code and the error message in that pastebin, it looks like the file has not been signed at all.

Andrei, which file exactly did you use? Where did you download it from? Can you send it to my email address?

Comment 44

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Andreas Wagner [:TheOne] from comment #43)
> Looking at the code and the error message in that pastebin, it looks like
> the file has not been signed at all.
> 
> Andrei, which file exactly did you use? Where did you download it from? Can
> you send it to my email address?

Andreas, the hotfix was fetched from the staging server, so Andrei has no file available. 
Felipe, maybe you can help Andreas?

Comment 45

[:Felipe Gomes (needinfo for replies!)]

Andreas, Kris signed it (from comment 39), and Andrei was getting it from the staging AMO server.

According to that log, the URL is https://addons-stage-cdn.allizom.org/user-media/addons/354399/mozilla_firefox_hotfix-20160826.01-fx.xpi?filehash=sha256%3A465d8f00d9ad113c1cc465e6dae3d4060e263f5f8c2e9e7ddcde868a8bff8d6b and I see that that file was signed.

Comment 46

[Andreas Wagner [:TheOne] [use NI]]

That file has been signed correctly. The fingerprint is "39:E7:2B:7A:5B:CF:37:78:F9:5D:4A:E0:53:2D:2F:3D:68:53:C5:60" which matches https://dxr.mozilla.org/mozilla-central/source/browser/app/profile/firefox.js#58

Comment 47

[Andreas Wagner [:TheOne] [use NI]]

Andrei, what is the value of "extensions.hotfix.certs.2.sha1Fingerprint" in your profile?

Comment 48

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Andreas Wagner [:TheOne] from comment #47)
> Andrei, what is the value of "extensions.hotfix.certs.2.sha1Fingerprint" in
> your profile?

I'll answer here since Andrei left for home (and I've also worked on this). The value is:
extensions.hotfix.certs.2.sha1Fingerprint;39:E7:2B:7A:5B:CF:37:78:F9:5D:4A:E0:53:2D:2F:3D:68:53:C5:60

Comment 49

[:Felipe Gomes (needinfo for replies!)]

Attachment: Unsigned xpi for signing

Fixed a syntax error

Comment 50

[Jason Thomas [:jason]]

Attachment: signed hotfix

Comment 51

[:Felipe Gomes (needinfo for replies!)]

Thanks Jason. Florin, I fixed a problem in the hotfix and just tested locally that it works. While it's not staged, there's a simpler way to test it:

- Download the signed hotfix from this bug
- Drag & drop it inside Firefox

You don't need any profile changes and you won't see the pref changes related to "hotfix", but you should still be able to verify that it's doing its intended effect (changing the "app.update.url" pref).


Btw, I found out what the issue with that log was, and it turns out the minimum version that this hotfix will work is Firefox 40 and not 39.  (bug 1151537 had an incorrect value for status-firefox39).

Comment 52

[Liz Henry (:lizzard) (relman/hg->git project)]

Will this also prevent affected users from releases earlier than 48, from updating to 49, once we ship 49?

Comment 53

[[e-mail] (:bhearsum)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #52)
> Will this also prevent affected users from releases earlier than 48, from
> updating to 49, once we ship 49?

The update rules can be configured that way, yes. Make sure that whomever is on Releaseduty is aware, and they'll make it happen.

Comment 54

[:Felipe Gomes (needinfo for replies!)]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #52)
> Will this also prevent affected users from releases earlier than 48, from
> updating to 49, once we ship 49?

That is more of a question for bug 1298503, as the hotfix itself is not controlling the update process, just adding the marks on the update ping. But yeah that's definitely doable with this.

Comment 55

[Andreas Wagner [:TheOne] [use NI]]

Comment on attachment 8785998 [details]
Unsigned xpi for signing

Felipe, as mentioned in IRC, if you want this up on stage, you need to increase the version number and we need to re-sign it. If going to prod directly (once QA approves) is fine, we can go ahead with the current file.

Comment 56

[:Felipe Gomes (needinfo for replies!)]

Yeah, if QA is happy with this one we can go to prod directly. If for whatever reason we need a new xpi I'll bump the version id before signing again.

Comment 57

[Matthew N. [:MattN]]

(In reply to :Felipe Gomes (needinfo me!) from comment #51)
> Thanks Jason. Florin, I fixed a problem in the hotfix and just tested
> locally that it works.
> …
> Btw, I found out what the issue with that log was, and it turns out the
> minimum version that this hotfix will work is Firefox 40 and not 39.  (bug
> 1151537 had an incorrect value for status-firefox39).

Hey Felipe, reminder to push the above changes to https://hg.mozilla.org/releases/firefox-hotfixes when you're happy with them.

Comment 58

[Bogdan Maris, Desktop Test Engineering]

We finished testing websense hotfix using .xpi attached in comment 50 using the following flow:

1. Started Firefox with a new profile.
2. Installed hotfix from comment 50

We used the following environment:

Platforms:
- Windows XP SP3 x86
- Windows 7 x64
- Windows 8.1 x86
- Windows 10 x64
- Mac OS X 10.10.5
- Ubuntu 16.04 x86

Firefox versions:
- Firefox 47
- Firefox 40
- Firefox 42
- Firefox 45
- Firefox 46

We successfully verified the following:

- Verified that the hotfix is indeed installed and the value of app.update.url contains (websense) and (nowebsense) when the .dll files are copied in ...\System32\, ...\sysnative\ and ...\SysWOW64\ (when needed).
- Verified that after update to FF 47.0.1 the value of 'app.update.url' remains the same (it contains websense and nowebsense) and the hotfix is still installed.
- Verified that on Mac OS X and Ubuntu the hotfix is installed and automatically uninstalled and also the value of 'app.update.url' remains untouched.

We also have a few Notes:

1. Installing the hotfix multiple times (I installed three times) will change the app.update.url value to: 
> https://aus4.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%(websense)(websense)(websense)/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

 - After update, only one (websense) is shown in the value of app.update.url.

2. Changing update channel to 'release-cdntest' will update to Firefox 48.0.2 no matter if the hotfix is installed and qipcap64.dll placed in C:\Windows\system32 and C:\Windows\SysWOW64\
 - app.update.url value after update: 
> https://aus5.mozilla.org/update/6/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%SYSTEM_CAPABILITIES%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

3. Updated from 40.0.3 to 43.0.1 to 47.0.1
- app.update.url changed from (40.0.3)
> https://aus4.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%(nowebsense)/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
- to (43.0.1)
> https://aus5.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%(nowebsense)/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

- to (47.0.1)
> https://aus5.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%(nowebsense)/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml


I don't think the above notes are any threat since they most likely are edgecases (#1.) or actually act as they should (#2. and #3.) but I thought that we should clear those things out just to be sure. 
Except for those notes all the test are a PASS from QA point of view.

For more informations regarding our testing please visit the following etherpad: https://public.etherpad-mozilla.org/p/websense-hotfix

Comment 59

[:Felipe Gomes (needinfo for replies!)]

Thanks. Yeah, looks like everything is fine!

#1 only happens due to installing multiple times from this manual process

#2 is expected that the hotfix will drop once 48 is reached

#3 part 1 were changes done in the app.update.url pref over time.

for part 2 ("to 47.0.1"): did you copy&paste the wrong value?  It looks equal as the one from 43.0.1 but I imagine it was just copy&paste mistake. It should start with "https://aus5.mozilla.org/update/6/"...

Comment 60

[:Felipe Gomes (needinfo for replies!)]

(In reply to Matthew N. [:MattN] from comment #57)
> Hey Felipe, reminder to push the above changes to
> https://hg.mozilla.org/releases/firefox-hotfixes when you're happy with them.

Already did, just forgot to post it here:
https://hg.mozilla.org/releases/firefox-hotfixes/rev/74417af85991

Comment 61

[Bogdan Maris, Desktop Test Engineering]

(In reply to :Felipe Gomes (needinfo me!) from comment #59)
> Thanks. Yeah, looks like everything is fine!
> 
> #1 only happens due to installing multiple times from this manual process
> 
> #2 is expected that the hotfix will drop once 48 is reached
> 
> #3 part 1 were changes done in the app.update.url pref over time.
> 
> for part 2 ("to 47.0.1"): did you copy&paste the wrong value?  It looks
> equal as the one from 43.0.1 but I imagine it was just copy&paste mistake.
> It should start with "https://aus5.mozilla.org/update/6/"...

After updating to 47.0.1 (from 43.0.1) the value is the one I mentioned in comment 58 (I just double checked):
https://aus5.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%(websense)/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml
Note that I only tried on Win XP to confirm this.

Comment 62

[:Felipe Gomes (needinfo for replies!)]

Ah right, thanks for confirmation. the version 6 of the url changed only on Firefox 48 (bug 1271761), so everything is right in Firefox 47.

Comment 63

[:Felipe Gomes (needinfo for replies!)]

Everything looks good for the hotfix. Andreas, can you push this hotfix to prod?

One detail: the install.rdf has minVersion=39.0, but can you configure AMO to ship it starting from version 40.0?

Comment 64

[Sylvestre Ledru [:Sylvestre]]

I sign off that we can go live now.

Comment 65

[Matthew N. [:MattN]]

(In reply to :Felipe Gomes (needinfo me!) from comment #59)
> #1 only happens due to installing multiple times from this manual process

Won't this also happen if we release an update to this hotfix since hotfix installs are restartless? We could easily fix this by adding the "/" after REPLACE_KEY in the search and replacement string. Otherwise we need to be aware of this potential future issue.

Comment 66

[:Felipe Gomes (needinfo for replies!)]

True. We could do that, but there are other ways to avoid it too if we need to release an update to it. I really don't want to have to re-sign and delay it further. Also, the OS_VERSION field is basically a free-form field so that's ok for the server.

Comment 67

[Andreas Wagner [:TheOne] [use NI]]

The hotfix version 20160826.01 has been deployed to prod: https://addons.mozilla.org/en-US/firefox/addon/firefox-hotfix/

Comment 68

[Jason]

You guys need to rethink how you distribute hotfixes.

We got approximately 350 calls this week about this hotfix, once we moved from 47.0.1 to 48.0.2.

Employees thought that they had some type of spyware, since they got an unfamiliar message about an extension they did not install.  If you’re going to deploy hotfixes, they need to be seamless and silent.

This can't happen again.  My hope is that this is just a one-time screwup, but it makes it really hard for IT to vouch for products like Firefox when there are visible boondoggles like this.

Comment 69

[Liz Henry (:lizzard) (relman/hg->git project)]

Jason: This was from bug 1300523, which should be fixed in 49.