1031516 - Continue (permanently) shipping the hotfix from bug 874513 to clients who haven't installed it yet

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Administration, defect, P2)

Description

[Benjamin Smedberg]

Older versions of Firefox expect a different signature for addon hotfixes than newer versions. This means that it is currently impossible to ship a single hotfix which applies "out of the box" to Firefox 10-24.

In bug 874513 we shipped hotfix "v20130826.01" which updated old installs to expect a new hotfix signing key. But with the release of more-recent hotfixes, this hotfix is no longer delivered. So people doing clean Firefox 10 installs (especially important for QAing new hotfixes) don't pass signature validation.

We don't know how many real users might not have installed the v20130826.01 hotfix, but Mossop says:

> D. From the server can we tell what certificate the client is expecting and give them a different .xpi? No
> 
> Actually although this is strictly true we can infer it. With the AMO check for
> a hotfix we do send the last hotfix version that the client installed. So if a
> Firefox 10-24 client makes a request with the last installed hotfix version
> <20130826.01 then they must be expecting the old certificate. I don't know if
> AMO supports this but in that case we could deliver the new cert hotfix.

This bug is to track our ability to continue delivering the old hotfix to these users, and newer hotfixes only after that one has been applied.

Comment 1

[Benjamin Smedberg]

Pauly in the meantime, in order to QA the update hotfix you'll need to manually set this pref:

extensions.hotfix.certs.1.sha1Fingerprint;91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA

Comment 2

[Wil Clouser [:clouserw]]

This is a very old service which we wanted to separate from the AMO code base entirely, but since it still has some includes from AMO it still lives there.  We should reevaluate if they are still necessary.  

Fortunately, it is an entirely standalone script, less than 400 lines, living at https://github.com/mozilla/olympia/blob/master/services/update.py .  An example URL is https://versioncheck-dev.allizom.org/update/VersionCheck.php?reqVersion=2&id=[e-mail]&version=&maxAppVersion=%ITEM_MAXAPPVERSION%&status=userEnabled,incompatible&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=%UPDATE_TYPE%&compatMode=%COMPATIBILITY_MODE%

Unfortunately, this bug is asking us to add a special case to it.  The GUID for the hotfix add-on is [e-mail].  This bug would involve looking for that GUID on the older Firefox version requests and responding with a hardcoded value if it's an old hotfix version.

As you can imagine, this URL gets *a lot* of traffic.  It's important that it remains as scalable as it is today so we can't do additional logic here and it needs to remain cached.

I'm marking this as a P2 because it has the potential of positively affecting a lot of users.  Let me know if there are questions/concerns.

Comment 3

[Paul Silaghi, QA [:pauly]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #1)
> Pauly in the meantime, in order to QA the update hotfix you'll need to
> manually set this pref:
> 
> extensions.hotfix.certs.1.sha1Fingerprint;91:53:98:0C:C1:86:DF:47:8F:35:22:
> 9E:11:C9:A7:31:04:49:A1:AA
Already changed the signature due to instructions on https://developer.mozilla.org/en-US/Add-ons/Hotfix#Testing_the_hotfix_on_the_staging_server, but still didn't work. I guess automatic install won't work until bug 1028388 gets fixed.

Comment 4

[Mathieu Agopian [:magopian]]

PR: https://github.com/mozilla/olympia/pull/136

This will need a thorough review.

Here's the rationale: if we receive a request for the "[e-mail]" addon
AND the reqVersion is less than 20130826.01, then, and only then, return versions that are less than or equal to 20130826.01.

We did this in order to keep the complex compat tests, and make sure we don't return a version that isn't correct.

Comment 5

[Mathieu Agopian [:magopian]]

Fixed in https://github.com/mozilla/olympia/commit/463098c4a72ea9d179118f21685cbebb7cf6f7ca

Please, let me know if you see this fix isn't working as expected!

Comment 6

[Benjamin Smedberg]

Awesome. Is this currently in staging or did it already hit production?

Pauly can you test this with stock Firefox 10 and see if we get the right updates?

Comment 7

[Mathieu Agopian [:magopian]]

This isn't in production yet, it'll be for the next push, in a few days

Comment 8

[Wil Clouser [:clouserw]]

It will need signoff from y'all and QA before it goes live.  It should be available today at versioncheck-dev.a.o (see full URL example in comment 0).

Comment 9

[Paul Silaghi, QA [:pauly]]

Still not working - FF 10, Win 7 x64:
STR:
1. Set extensions.logging.enabled to true
2. Set extensions.update.interval to 10
3. changed 'versioncheck.addons.mozilla.org' with 'versioncheck-dev.allizom.org' in the 'extensions.update.url' pref
4. set extensions.hotfix.certs.1.sha1Fingerprint=91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA
5. executed the ping:
var obj = {};
Cu.import("resource://gre/modules/AddonManager.jsm", obj);
obj.AddonManagerPrivate.backgroundUpdateCheck();

Error console results:
"Error: Expected certificate attribute 'sha1Fingerprint' value incorrect, expected: '91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA', got: 'CA:C4:7D:BF:63:4D:24:E9:DC:93:07:2F:E3:C8:EA:6D:C3:94:6E:89'.
Source File: resource://gre/modules/CertUtils.jsm
Line: 103

Error: Certificate checks failed. See previous errors for details.
Source File: resource://gre/modules/CertUtils.jsm
Line: 106

LOG addons.xpi: Cancelling download of https://addons-dev-cdn.allizom.org/storage/public-staging/354399/mozilla_firefox_hotfix-20130826.01-fx.xpi"


Same results with 'addons-dev.allizom.org' instead of 'versioncheck-dev.allizom.org' at step 3.
Note that the signature is set correctly in the step 4, it's not the wrong one displayed in the error console.

Comment 10

[Mathieu Agopian [:magopian]]

:pauly, when you say the signature is set correctly in step 4, I'm pretty sure it's not the "correct" signature, if I understand things correctly.

If you're using the new signature to install the 20130826.01 hotfix (which is the one in charge of changing the old signature to the new signature), it won't work. The 20130826.01 hotfix should be signed with the old signature (unless I missed something in the description of this bug), and will update the signature to the new signature (so any hotfix coming after this one should be signed with the new signature).

I believe there's two scenarios to check:
1/ a version of firefox with the old signature, and an old version of the hotfix, requesting an update: in this case, the server should reply with the 20130826.01 hotfix (and not the newest one: 20140319.01, which is signed with the new signature). Once this 20130826.01 hotfix is installed, the signature should match the new signature ('91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA')
2/ a version of firefox with the new signature (like the one you should have at the end of scenario 1/), and at least the 20130826.01 version of the hotfix (or above). The server in this case should reply with the 20140319.01 hotfix, and it should install successfully.

Please let me know if I misunderstood something, or if this explanation isn't clear enough.

Comment 11

[Paul Silaghi, QA [:pauly]]

Attachment: FF 28 update log.txt

Ok, I've tried now without changing the signatures, but still without success:
FF 10:
Error: Expected certificate attribute 'sha1Fingerprint' value incorrect, expected: 'F1:DB:F9:6A:7B:B8:04:FA:48:3C:16:95:C7:2F:17:C6:5B:C2:9F:45', got: 'CA:C4:7D:BF:63:4D:24:E9:DC:93:07:2F:E3:C8:EA:6D:C3:94:6E:89'.
Source File: resource://gre/modules/CertUtils.jsm
Line: 103
LOG addons.xpi: Cancelling download of https://addons-dev-cdn.allizom.org/storage/public-staging/354399/mozilla_firefox_hotfix-20130826.01-fx.xpi

FF 28:
see attached
btw, I'm running FF 28 en-US, in the log I see es-MX, which I also have it installed, but it's not running

Comment 12

[Mathieu Agopian [:magopian]]

According to https://bugzilla.mozilla.org/show_bug.cgi?id=874513#c24, the signature/fingerprint should be CA:C4:7D:BF:63:4D:24:E9:DC:93:07:2F:E3:C8:EA:6D:C3:94:6E:89, not F1:DB:F9:6A:7B:B8:04:FA:48:3C:16:95:C7:2F:17:C6:5B:C2:9F:45.

I don't know why you're having this other signature/fingerprint. Maybe it's because this signature/fingerprint was changed at another time (according to https://developer.mozilla.org/en-US/Add-ons/Hotfix#Signatures_2, the signature/fingerprint was already changed following https://bugzilla.mozilla.org/show_bug.cgi?id=803596?).

Does that mean we need two special cases? One for the "very old" signature, and one for the "just old" signature?

I need more info on that one, [:bsmedberg] maybe?

Comment 13

[Paul Silaghi, QA [:pauly]]

(In reply to Paul Silaghi, QA [:pauly] from comment #11)
> FF 28:
> see attached
> btw, I'm running FF 28 en-US, in the log I see es-MX, which I also have it
> installed, but it's not running
The update doesn't work in this case too. The hotfix is not installed, nothing happens after restarting FF.

Comment 14

[Benjamin Smedberg]

I was not aware of the second signature change :-(

This means that users of Firefox 10-16 will first need to get the v20120817.01 hotfix, and then v20130826.01? Felipe can you confirm that this is the correct sequence?

Comment 15

[:Felipe Gomes (needinfo for replies!)]

Yeah (I had mentioned it in that email thread :)   But the first hotfix to use is v20121019.01, not v20120817.01.

v20121019.01 was a combined hotfix that included the the two previous hotfixes (bug 774509 and bug 790096), plus the cert update (bug 803596)

Also, the cert update was included in a chemspill on FF 16.0.2.  So only users up to FF 16.0.1 will need the first one. bug 803583 comment 14

Comment 16

[Wil Clouser [:clouserw]]

Sorry, this is getting confusing for people who aren't very familiar with the hotfixes.  Would someone please clarify exactly what should happen at this point so we can adjust the code?  pseudo-code encouraged! :)

Comment 17

[Benjamin Smedberg]

Felipe can you say which exact versions of Firefox should get which hotfixes?

Comment 18

[Mathieu Agopian [:magopian]]

Maybe that's not the correct way to do things, but atm, we're only checking on the installed version of the hotfix, not the FF version. Should we also check the FF version?

Please review the rationale of the current fix in comment 4

Comment 19

[:Felipe Gomes (needinfo for replies!)]

FF 10 - 16.0.1 =>
  need to first receive v20121019.01
  then receive v20130826.01

FF 16.0.2 - FF 24.*
  need to only receive v20130826.01


Just reiterating that v20121019.01 contains bug 774509 and bug 790096

Comment 20

[Mathieu Agopian [:magopian]]

Second attempt at fixing the bug: https://github.com/mozilla/olympia/pull/140

Comment 21

[Mathieu Agopian [:magopian]]

(re)fixed in https://github.com/mozilla/olympia/commit/efc6dc8c244d81bfff1e37fc52d00395211b7b73

[:pauly], please let me know if this still fails

Comment 22

[Paul Silaghi, QA [:pauly]]

Attachment: FF 10 update log.txt

I don't see any update working.

Comment 23

[Mathieu Agopian [:magopian]]

I need help to understand the log.

It seems your Firefox is properly requesting for an update, and properly receiving the answer that it should download 20121019.01

It then downloads it, installs it (and then there's a warning about '[e-mail]' being an unknown addon-ID? Does that mean this addon wasn't installed after all?).

It then asks for an update again, but specifies '2' instead of '20121019.01' for the reqVersion, which is strange...

I don't know why it's not requesting anything after the restart, is that normal?

Comment 24

[Paul Silaghi, QA [:pauly]]

(In reply to Mathieu Agopian [:magopian] from comment #23)
> I need help to understand the log.
me too

Comment 25

[Benjamin Smedberg]

I'll defer to Felipe.

Comment 26

[Dave Townsend [:mossop]]

(In reply to Mathieu Agopian [:magopian] from comment #23)
> I need help to understand the log.
> 
> It seems your Firefox is properly requesting for an update, and properly
> receiving the answer that it should download 20121019.01
> 
> It then downloads it, installs it (and then there's a warning about
> '[e-mail]' being an unknown addon-ID? Does that mean this
> addon wasn't installed after all?).

That warning as I recall is ignorable, it's just to do with some of the bootstrap code in that add-on.

> It then asks for an update again, but specifies '2' instead of '20121019.01'
> for the reqVersion, which is strange...

reqVersion is always 2. version on the other hand correctly says that it previously installed 20121019.01.

So everything works fine but when it attempts to check for a new hotfix after installing 20121019.01, the update service still tells it that the newest version is 20121019.01.

Comment 27

[Dave Townsend [:mossop]]

(In reply to Mathieu Agopian [:magopian] from comment #21)
> (re)fixed in
> https://github.com/mozilla/olympia/commit/
> efc6dc8c244d81bfff1e37fc52d00395211b7b73
> 
> [:pauly], please let me know if this still fails

Mathieu, reqVersion is not the correct parameter to check for the hotfix version. You want the version parameter.

Comment 28

[Mathieu Agopian [:magopian]]

I see, fixed in PR https://github.com/mozilla/olympia/pull/145

Comment 29

[Mathieu Agopian [:magopian]]

Fixed (hopefully) in https://github.com/mozilla/olympia/commit/958faa18cf255697d0f6c411446309944419f593

:pauly, can you please confirm?

Comment 30

[Benjamin Smedberg]

Pauly is on PTO this week, Florin is filling in.

Comment 31

[Catalin Varga [QA][:VarCat]]

I've tested the hotfix update using FF 24 and FF10 on Win 7 x64 with the following steps:

STR:
1. Set extensions.logging.enabled to true
2. Set extensions.update.interval to 10
3. changed 'versioncheck-bg.addons.mozilla.org' with 'addons-dev.allizom.org' in the 'extensions.update.background.url' pref.
4. Restart Firefox.

Behavior noticed:
 1. extensions.update.interval parameter is not working properly, setting it to 10 has various behaviors sometimes the addons.updates it triggered almost immediately and in other instances I waited 30 minutes without it being triggered. Due to this issue the testing was done using the script:

"var obj = {};
Cu.import("resource://gre/modules/AddonManager.jsm", obj);
obj.AddonManagerPrivate.backgroundUpdateCheck();"

2. Running that script once will download an older version of hotfix-update(20130826.01 version for FF 24 ) addon that will be installed, started, stopped and uninstalled.  
Running the script again will find and update for the hotfix-update(20140527.01.1 version) download, install, starting, successfully run it but the addon will not be uninstalled at the end.

There is a similar behavior for FF10 with some differences:

3. There is an error in the logs
Error: ERROR addons.xpi: Failed to remove file C:\Users\catalin.varga\AppData\Roaming\Mozilla\Firefox\Profiles\qj8fx1h0.jjjjj\extensions\trash\[e-mail]: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.remove]"  nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)"  location: "JS frame :: resource:///modules/XPIProvider.jsm :: recursiveRemove :: line 1210"  data: no]
Source File: resource:///modules/XPIProvider.jsm
Line: 1210

4. The older hotfix-update addon version is different from the one on FF24 eg 20121019.01 but that's expected.

5. Updating the signature prior to the installation of the hotfix-update addon will prompt this error:
Error: Expected certificate attribute 'sha1Fingerprint' value incorrect, expected: '91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA', got: 'CA:C4:7D:BF:63:4D:24:E9:DC:93:07:2F:E3:C8:EA:6D:C3:94:6E:89'.

Conclusions:
1. hotfix is working but it requires 2 addons.updates triggers the first one to download the older version of the hotfix and the second one to download the new version.
2. extensions.update.interval parameter is highly unreliable.
3. The signature must not be changed for older FF versions, prior to the hotfix addon installation or the installation will fail.

Further testing should be done on other operating systems and with other scenarios.

Comment 32

[Benjamin Smedberg]

Catalin when do you expect the final signoff for this?

Comment 33

[Catalin Varga [QA][:VarCat]]

The final testing can be covered in 2-3 days, a possible date for sign off was provided by Florin Mezei via email, but to proceed further I need feedback on the behavior described in comment 31 .

Comment 34

[Mathieu Agopian [:magopian]]

From what I understand, the fix seems to now be working correctly. However, I also see many possible side effects and implications that I don't understand, so I don't feel comfortable pushing that to production (the next push is in 6 hours).

If you also feel the fix in this bug is now correct and should go to production, please let me know in the mean time!

Comment 35

[Benjamin Smedberg]

What feedback do you need on comment 31? What you described sounds like this is working correctly.

Comment 36

[Catalin Varga [QA][:VarCat]]

The discussion was back and forth and several issues were raised so the behavior was no exactly clear to me. I needed a confirmation that this was the correct behavior. We will test using FF 10-28 on Win 7, Win Xp, Win 8.1 and Win Vista.

Comment 37

[Benjamin Smedberg]

> Running the script again will find and update for the
> hotfix-update(20140527.01.1 version) download, install, starting,
> successfully run it but the addon will not be uninstalled at the end.

To be clear, this is the update hotfix. This hotfix downloads and installs Firefox 30, and so it won't uninstall immediately: it should only uninstall after it has successfully downloaded and upgraded to Firefox 30.

So we should (in the update hotfix bug) make sure that it is functioning correctly, but in terms of this bug, the AMO updates are being served correctly and in the correct order.

Mathieu, I think this is ready to ship.

Comment 38

[Mathieu Agopian [:magopian]]

Excellent news, thanks, it'll go in production today then.

Comment 39

[Wil Clouser [:clouserw]]

This went live.

Comment 40

[Paul Silaghi, QA [:pauly]]

Attachment: FF 10 update log.txt

This is working on staging, but not in production

Comment 41

[Paul Silaghi, QA [:pauly]]

Attachment: FF 28 update log.txt

Comment 42

[Paul Silaghi, QA [:pauly]]

Note that I executed the code snippet more than 3 times, according to https://bugzilla.mozilla.org/show_bug.cgi?id=1034216#c17

Comment 43

[Wil Clouser [:clouserw]]

I confirmed with ops that it's live (tag @ https://github.com/mozilla/olympia/commits/2014.07.10)

Comment 44

[Benjamin Smedberg]

In production, requesting the URL https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=[e-mail]&version=20130826.01&maxAppVersion=%ITEM_MAXAPPVERSION%&status=userEnabled,incompatible&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=10.0&appOS=WINNT&appABI=x86-msvc&locale=en-US&currentAppVersion=10.0&updateType=%UPDATE_TYPE%&compatMode=%COMPATIBILITY_MODE%

returns mozilla_firefox_hotfix-20130826.01-fx.xpi. I believe that's correct, because the only newer hotfix in production is v20140319.01, which has a minVersion of 27.0. So I believe this code is working correctly.

The v20140527.01 update hotfix is staged but hasn't been pushed to production yet.

Comment 45

[Paul Silaghi, QA [:pauly]]

Verified fixed based on comments 40, 41 and 44.