874513 - Create an add-on hotfix to update the hotfix cert for Firefox 10-24

Status: VERIFIED FIXED

(Firefox :: General, defect)

Description

[Alex Keybl [:akeybl]]

Assuming bug 870329 lands in FF24, we'll need to create an add-on hotfix to update the hotfix cert for Firefox 10-23 before October 2014.

Putting this on the tracking nom list because we don't have a longer term tracking process (yet).

Comment 1

[Alex Keybl [:akeybl]]

(In reply to Alex Keybl [:akeybl] from comment #0)
> Assuming bug 870329 lands in FF24, we'll need to create an add-on hotfix to
> update the hotfix cert for Firefox 10-23 before October 2014.
> 
> Putting this on the tracking nom list because we don't have a longer term
> tracking process (yet).

I think I meant to say assuming bug 870329 *doesn't* land.

Comment 2

[bhavana bajaj [:bajaj]]

Adding needsinfo on :Mossop here to see if he can help with this request ?

Comment 3

[Dave Townsend [:mossop]]

I don't have time to help to write this. It should be fairly simple though, just changing a pref. We've done it before, though I think it was mixed in with a bunch of other fixes too.

Comment 4

[Alex Keybl [:akeybl]]

Felipe, Gavin suggested you'd be able to help here.

bhearsum, when do we expect to have the new cert in hand to give to Felipe?

Comment 5

[:Felipe Gomes (needinfo for replies!)]

Sure, I'll take it. There is one thing we need to decide: when we release this hotfix, users who have not yet received the previous hotfix will skip through it and just receive the most recent version, meaning that they will not get bug 812573.

We have two options here:
- if we believe that most users who were supposed to receive bug 812573 have already received it, we can call it not a problem and take the simpler approach to just ship the new hotfix to update the cert.
- otherwise, we can include in the new hotfix the code to do the two things at once for people who receive it: updating the cert for everyone and updating the update interval for fx10-16.

Code wise there's barely any difference for me as we can just reuse 99% the code from bug 812573, so the main difference would be in the release process. (mostly i'm thinking that the latter will require more testing, etc. ). So I'll wait for you to make the call

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Alex Keybl [:akeybl] from comment #4)
> Felipe, Gavin suggested you'd be able to help here.
> 
> bhearsum, when do we expect to have the new cert in hand to give to Felipe?

I'm not planning on renewing until September.

Comment 7

[bhavana bajaj [:bajaj]]

:bhearsum, when does the current fingerprint expire ? We release Fx24 on Sep 17 and goto build with our release on Sep 9th with our final beta/release candidate.

I would ideally like to have the cert in Felipe's hand early enough in Sep so he has a few days in hand and can target Beta 9 which goes to build on Sep 5th . Does that sound feasibile ?

Comment 8

[[e-mail] (:bhearsum)]

(In reply to bhavana bajaj [:bajaj] from comment #7)
> :bhearsum, when does the current fingerprint expire ?

The current certificate expires on October 18th.

> I would ideally like to have the cert in Felipe's hand early enough in Sep
> so he has a few days in hand and can target Beta 9 which goes to build on
> Sep 5th . Does that sound feasibile ?

Why does this need to ship with the release? Even if it does, we need to release a hotfix with the updated fingerprints to make sure that users who don't update to 24 before October 18th receive the new fingerprints.

I don't think I can hit September 5th. I'm fully offline two days next week and as part of the renewal I'm planning to look at switching CAs, which I haven't done any research on yet.

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Attachment: Update cert fingerprint (w/ placeholder cert)

I created the hotfix with a placeholder cert so that we can already start the QA process whenever desired, and just swap in the string for the real cert when we get it.

It targets all platforms, 10.0 - 23.*

(I also updated the icons in the add-on with the new Firefox logo)


Regarding my previous question in comment 5, I think the last hotfix was released long enough that we don't need to merge the previous hotfix in here.


(In reply to Alex Keybl [:akeybl] from comment #0)
> Assuming bug 870329 lands in FF24, we'll need to create an add-on hotfix to
> update the hotfix cert for Firefox 10-23 before October 2014.

Just curious, since it seems we want to have the cert updated for FF24, did you mean before Oct `2013`?

Comment 10

[[e-mail] (:bhearsum)]

Bhavana and I talked about this today to try and clear up confusion. She was particularly worried about our ability to ship a hotfix to users after the current cert expires (October 17th) but before Firefox 25 ships (October 29th). This is not an issue because we should be shipping out a hotfix with the updated fingerprints after I renew the cert (sometime mid-late september).

The reason I'm not pushing to do this renewal ahead of Firefox 24 is because if we do need to ship a hotfix between October 17th and October 29th, we would need to ship new fingerprints in a hotfix _anyways_, because our uptake rate on actual updates isn't fast enough to protect a large portion of our userbase.

So to be clear, these are my expectations:
* Mid/late September I will renew our Authenticode certificate and provide new fingerprints that Firefox should use to validate updates.
* Somebody needs to update the fingerprints in our repositories (central, aurora, beta, release, esr17, esr24).
* Somebody needs to finish creating the hotfix that updates the fingerprints for existing Firefox users.
* I will sign the hotfix that updates the fingerprints.
* Somebody needs to ship the hotfix.
* Any releases generated after the certificate has been renewed (mid/late cycle 25.0 betas, 25.0 release, any point releases that may happen between renewal and 25.0 being shipped) will be signed by the renewed certificate.

I hope this clears things up, and please let me know if you have any questions or concerns!

Comment 11

[:Felipe Gomes (needinfo for replies!)]

Attachment: hotfix-v20130826.01.xpi

Uploading the built hotfix for testing

Comment 12

[:Felipe Gomes (needinfo for replies!)]

Attachment: Update cert fingerprint (w/ placeholder cert), 10.0 - 24.*

Just putting it on the radar. Bumped max version to 24.* given comment 10.

Comment 13

[:Felipe Gomes (needinfo for replies!)]

Attachment: hotfix-v20130826.01.xpi, 10.0 - 24.*

This xpi has maxVersion bumped to 24.*

Comment 14

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Comment on attachment 796761 [details] [diff] [review]
Update cert fingerprint (w/ placeholder cert), 10.0 - 24.*

Review of attachment 796761 [details] [diff] [review]:
-----------------------------------------------------------------

::: README
@@ +28,5 @@
>                 it in the future, it needs to have its version id (folder name) bumped.
>  v20130322.01 - Bug 812573 - Decrease the update interval for pre-17 users of beta, esr
>                 and release channels using mozilla.org servers in all platforms.
> +v20130826.01 - Bug 874513 - A hotfix to update the hotfix certificate fingerprint,
> +               rolled out to 10.0 - 23.*.

Version here needs updated.

Comment 15

[[e-mail] (:bhearsum)]

New fingerprints are: 91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA

I'm updating the necessary repositories as part of bug 803531, ftr.

Comment 16

[:Felipe Gomes (needinfo for replies!)]

Attachment: Update cert fingerprint (w/ correct cert), 10.0 - 24.*

Patch pushed to the repo, using the fingerprint from comment 15 here (and bug 803531)

Comment 17

[:Felipe Gomes (needinfo for replies!)]

Attachment: hotfix-v20130826.01.xpi, 10.0 - 24.*, unsigned

Unsigned hotfix. It can also be generated by running:

HOTFIX=v20130826.01 make package

in the root of the hotfixes repo

Comment 18

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/firefox-hotfixes/rev/40a29960eaef

Comment 19

[[e-mail] (:bhearsum)]

Attachment: hotfix signed with old keys

Here's a copy of the hotfix signed with the _old_ keys. Who's responsible for shipping this out? Is this happening as part of this bug, or elsewhere?

Comment 20

[:Felipe Gomes (needinfo for replies!)]

I think the release can be handled in this bug (or a different one if someone prefers), but the QA we need to do first should happen on a separate bug (this was asked by QA the last time we did a hotfix). So I'll file a bug for that.

Jorge: can you upload this signed hotfix to the staging server, like you did in bug 865873 comment 3?

Comment 21

[Tracy Walker [:tracy]]

I don't care if the QA is done here or not, I just need clear instructions with steps to follow and expected results.  Something along the lines of https://wiki.mozilla.org/QA/Desupport_Hotfix_Testing#Instructions_to_test_a_hotfix would be very helpful.  I'm not sure what this hotfix is doing, so can't say if that sort of plan is overkill or not.

Comment 22

[Jorge Villalobos [:jorgev] (he/him)]

The new version is up on staging now. It should take an hour or so for it to be fully available.

Comment 23

[Tracy Walker [:tracy]]

Please explain how to confirm this hotfix does what it is supposed to, then I'll verify against stage.

Comment 24

[Tracy Walker [:tracy]]

I used the instructions at https://wiki.mozilla.org/QA/Desupport_Hotfix_Testing#Installing_Hotfix to check for the hotfix against stage.

I assume I'm looking for extensions.hotfix.certs.1.sha1Fingerprint;CA:C4:7D:BF:63:4D:24:E9:DC:93:07:2F:E3:C8:EA:6D:C3:94:6E:89 to be updated to the new fingerprint, in addition to the hotfix add-on presence in add-ons manager.

No change and no hotfix add-on. In logs,I am getting:

[09:47:30.879] LOG addons.updates: Requesting https://addons-dev.allizom.org/update/VersionCheck.php?reqVersion=2&id=[e-mail]&version=&maxAppVersion=%ITEM_MAXAPPVERSION%&status=userEnabled,incompatible&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=%UPDATE_TYPE%&compatMode=%COMPATIBILITY_MODE%
[09:47:30.938] GET https://addons-dev.allizom.org/update/VersionCheck.php?reqVersion=2&id=[e-mail]&version=&maxAppVersion=%ITEM_MAXAPPVERSION%&status=userEnabled,incompatible&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=%UPDATE_TYPE%&compatMode=%COMPATIBILITY_MODE% [HTTP/1.1 301 Moved Permanently 196ms]
[09:47:30.892] LOG addons.repository: Requesting https://services.addons.mozilla.org/en-US/firefox/api/1.5/search/guid:%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D?src=firefox&appOS=Darwin&appVersion=24.0&tMain=22&tFirstPaint=730&tSessionRestored=755
[09:47:30.940] GET https://services.addons.mozilla.org/en-US/firefox/api/1.5/search/guid:%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D?src=firefox&appOS=Darwin&appVersion=24.0&tMain=22&tFirstPaint=730&tSessionRestored=755 [HTTP/1.1 200 OK 153ms]
[09:47:31.137] GET https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=[e-mail]&version=&maxAppVersion=%ITEM_MAXAPPVERSION%&status=userEnabled,incompatible&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=%UPDATE_TYPE%&compatMode=%COMPATIBILITY_MODE% [HTTP/1.1 200 OK 248ms]
[09:47:31.111] LOG addons.updates: Requesting https://addons-dev.allizom.org/update/VersionCheck.php?reqVersion=2&id={972ce4c6-7e08-4474-a285-3208198ce6fd}&version=24.0&maxAppVersion=24.0&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=112&compatMode=normal
[09:47:31.138] GET https://addons-dev.allizom.org/update/VersionCheck.php?reqVersion=2&id={972ce4c6-7e08-4474-a285-3208198ce6fd}&version=24.0&maxAppVersion=24.0&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=112&compatMode=normal [HTTP/1.1 301 Moved Permanently 65ms]
[09:47:31.235] GET https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id={972ce4c6-7e08-4474-a285-3208198ce6fd}&version=24.0&maxAppVersion=24.0&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=24.0&appOS=Darwin&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=24.0&updateType=112&compatMode=normal [HTTP/1.1 200 OK 239ms]
[09:47:31.426] WARN addons.updates: Update manifest for {972ce4c6-7e08-4474-a285-3208198ce6fd} did not contain an updates property @ resource://gre/modules/AddonUpdateChecker.jsm:312

Comment 25

[:Felipe Gomes (needinfo for replies!)]

It seems that the request for the hotfix update in the staging server is returning as no update available. I'm currently investigating if the request is not correct or if it's a cache problem in the staging server.

Meanwhile, you can test the add-on manually on different versions by downloading the signed xpi and drag&dropping it into Firefox. You should see a message that the add-on was installed, the sha1Fingerprint pref updated, and no traces of the hotfix afterwards as it will uninstall itself immediatelly.
(If you have the add-ons manager tab open while doing it it's possible that it won't disappear from the list until you close the tab and re-open it)

The manual testing won't update the extensions.hotfix.lastVersion pref

Comment 26

[Tracy Walker [:tracy]]

ok, manual check confirms the add-on updated the fingerprint to 91:53:98:0C:C1:86:DF:47:8F:35:22:9E:11:C9:A7:31:04:49:A1:AA.

let me know when you want me to try against stage again.

Comment 27

[Tracy Walker [:tracy]]

This looks good to go to production.

Comment 28

[Jorge Villalobos [:jorgev] (he/him)]

The new hotfix version is now live in prod.

Comment 29

[[e-mail] (:bhearsum)]

Thanks everyone!

Comment 30

[Tracy Walker [:tracy]]

Was chasing a topcrash issue yesterday. A day late to verify here, but this looks good live with a few spot checks.

Comment 31

[:Felipe Gomes (needinfo for replies!)]

I created a page on MDN to document the hotfix process and testing for easier future reference, since the info was scattered through some bugs and wiki pages:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/Hotfix

I also checked today a friend's computer who is running Firefox 24 release, and they have already received this hotfix.