913918 - we may need to hotfix app.update.certs.3.* for digicert

Status: VERIFIED WONTFIX

(Firefox :: General, defect)

Description

[Alex Keybl [:akeybl]]

See https://bugzilla.mozilla.org/show_bug.cgi?id=912675#c7

We would likely want this hotfix for all Firefox versions back to 10, on all platforms.

+++ This bug was initially created as a clone of Bug #912675 +++

Greetings,

The current Firefox is pinned to accept only either Equifax or Thawte for SSL certificates on aus3.mozilla.org (for updating). Equifax is in use right now, and Thawte is a backup.

This cert is about to expire and we'll have to renew, but we don't yet know which of those 2 vendors we can renew with. So, we don't want to change either of them at this point.

We do, however, want to add Digicert as an allowed vendor. They are our new/current SSL certificate vendor.

The new entries would be:

app.update.certs.3.commonName
aus3.mozilla.org

app.update.certs.3.issuerName
C=US; O=DigiCert Inc; CN=DigiCert Secure Server CA

We'd like to do this as soon as possible (uplift to Aurora/Beta even, so it goes out on next release)... the sooner it's available on release, the sooner it becomes realistically feasible to start using Digicert (we have to wait for a significant level of penetration, so we can't go right away... an extra 2-3 versions would help).

Comment 1

[Robert Strong (they/them - no direct email)]

Since this is for an add-on hotfix, is likely critical, and the fact that the add-on hotfix won't fix everyone I very much think we should also add a new aus host that serves up the same content, update the client's app.update.url to point to the new host, and add the new cert to that server. This way clients that don't get the add-on hotfix will still be able to update using aus3 along with the old cert that has been updated and clients that get the update can use the new host along with the new cert. If we kept the aus3 app.update.url then the new cert pref values would only provide value if we don't get the cert renewed in that the clients that get the new values would be able to update though this would of course prevent clients that don't get the new cert pref values from updating or if we decided to cutover to the digicert cert sooner rather than later in which case once again clients that don't get the updated cert pref values once again wouldn't be able to update.

The exception to this is if you could somehow use multiple certificates on the same host which I don't think is possible.

Comment 2

[Jake Maul [:jakem]]

(In reply to Robert Strong [:rstrong] (do not email) from comment #1)
> Since this is for an add-on hotfix, is likely critical, and the fact that
> the add-on hotfix won't fix everyone I very much think we should also add a
> new aus host that serves up the same content, update the client's
> app.update.url to point to the new host, and add the new cert to that
> server. This way clients that don't get the add-on hotfix will still be able
> to update using aus3 along with the old cert that has been updated and
> clients that get the update can use the new host along with the new cert.

Unless Firefox will accept an expired certificate, this won't help.

*If* the Thawte certificate on order arrives in time, and *if* the immediate signing cert has the identification string that Firefox is pinned to, then we will definitely use that certificate- this bug won't matter and could be WONTFIX'd at that point. The first "if" doesn't bother me much, I'm likely to have the cert sometime tomorrow. The second "if" is much more worrisome- I can't verify that until I have it, and if it's not the expected value there's nothing I can do about it. We must plan for that case, and therefore I believe it's prudent to proceed with work on the addon hotfix even before we know if we'll need it. I don't see how it can hurt (except for wasting some time if we don't need it urgently after all), and it definitely gives us an extra way out.

To reiterate: if we *can* use the on-order Thawte cert then we will, and this addon will have been unnecessary and aus3 will continue working as normal. If we can't, then it doesn't help to put Digicert on a new name... the old name/cert is going to be broken anyway.

There is no case where we would want some users to get a working Digicert property and others to get a working Thawte property. If the latter works, everyone will use it. If not, everyone that can be hotfixed will use the former, and everyone that can't will be out in the cold.

I can't see any situation where it helps to put Digicert on a new name. Changing app.update.url feels like extra risk (and extra work) for no gain.


The only question I see is: are we willing to delay this addon until we know for sure we'll need it? I don't know how easy or hard it is to make/deliver it, or how fast we can expect uptake to be, so I can't answer this. If the answers are "simple" and "very fast", perhaps we can wait.


> If
> we kept the aus3 app.update.url then the new cert pref values would only
> provide value if we don't get the cert renewed in that the clients that get
> the new values would be able to update though this would of course prevent
> clients that don't get the new cert pref values from updating or if we
> decided to cutover to the digicert cert sooner rather than later in which
> case once again clients that don't get the updated cert pref values once
> again wouldn't be able to update.

If we can get a working/usable Thawte cert, then I have no intention of cutting over to Digicert any time soon. The cert on order is valid for 4 years.

Digicert (and this bug) is a contingency plan only.

Comment 3

[Robert Strong (they/them - no direct email)]

That isn't the full story and I replied in the thread on release drivers.

Comment 4

[Matthew N. [:MattN]]

I can take this. I'll make two hotfixes (with and without the update URL change). I may need more details as I dig into this further.

Comment 6

[Blair McBride [:Unfocused] (UNAVAILABLE)]

(In reply to Matthew N. [:MattN] from comment #4)
> I can take this. I'll make two hotfixes (with and without the update URL
> change). I may need more details as I dig into this further.

Since I can't get you on IRC, I assume you're already head-down and busy working on this. If so, ping me for review/feedback/questions. If not, it's the middle of the day here and I'm happy to take it.

Comment 7

[Matthew N. [:MattN]]

(In reply to Blair McBride [:Unfocused] from comment #6)
> Since I can't get you on IRC, I assume you're already head-down and busy
> working on this. If so, ping me for review/feedback/questions. If not, it's
> the middle of the day here and I'm happy to take it.

I ran out to get dinner before I started. You can take this :) Thanks

Comment 8

[Jake Maul [:jakem]]

(In reply to Robert Strong [:rstrong] (do not email) from comment #3)
> That isn't the full story and I replied in the thread on release drivers.

I've read up on that and replied there. Short answer is, rstrong's plan has one key advantage in that, if for some reason we cannot renew "aus3.mozilla.org" on Thawte or Equifax, then we can hotfix-move as many people as possible to "aus3-1.mozilla.org" on Digicert, and shut down aus3.mozilla.org altogether. Anyone that doesn't get moved will get a reasonable fallback that directs them to some sort of full installer (www.mozilla.org, I guess). There's even an outside chance that some really old installs might get updated due to this, if they're still trying (but failing) to update.

I'm willing to set up the server side for this. I'm pretty sure we can have it done tomorrow, unless there's yet another aspect of this system I don't understand. :)


Another piece of the puzzle, perhaps out of scope here but worth mentioning: Thunderbird, Seamonkey, and Metro Firefox apparently do not support addon hotfixing, but (AIUI) are still pinned. If we can't fix aus3.mozilla.org to the satisfaction of what's already out there, we're basically SOL on them... whether it's up or down, they won't be able to update. I don't know if we can do anything more than to try and spin a release on them ASAP.

Comment 9

[Robert Kaiser]

(In reply to Jake Maul [:jakem] from comment #8)
> Another piece of the puzzle, perhaps out of scope here but worth mentioning:
> Thunderbird, Seamonkey, and Metro Firefox apparently do not support addon
> hotfixing, but (AIUI) are still pinned.

SeaMonkey is running off the aus2-community server with a different cert and had its pinning updated in bug 679677 - that doesn't include digicert but geotrust, which is what is in use there at this time, AFAIK. In any case, a separate issue.

Thunderbird and Metro run off the same server as Firefox, Metro hasn't been shipped in release though, so a fix in beta would be pretty efficient there.

Comment 10

[Blair McBride [:Unfocused] (UNAVAILABLE)]

Attachment: Patch v1

Comment 11

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 801551 [details] [diff] [review]
Patch v1

Review of attachment 801551 [details] [diff] [review]:
-----------------------------------------------------------------

So, just to spell it out clearly as it differs from comment 0, the pref changes meant to be done by this hotfix are:

change app.update.url to "https://aus3-1.mozilla.org/" + ...

add app.update.certs.3.commonName as "aus3-1.mozilla.org"
add app.update.certs.3.issuerName as "CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US"
-------


Would there be any value in using maxVersion=26.* instead of maxVersion=26.0a1?

Should also add an entry in the README file about this hotfix

::: v20130909.01/bootstrap.js
@@ +41,5 @@
> +    Services.prefs.setCharPref(UPDATE_CERTS_ISSUERNAME_PREF, CERT_ISSUERNAME);
> +
> +    let updateUrl = Services.prefs.getCharPref(UPDATE_URL_PREF);
> +    updateUrl = UPDATE_URL_NEW_HOST + updateUrl.substr(UPDATE_URL_OLD_HOST.length);
> +    Services.prefs.setCharPref(UPDATE_URL_PREF, updateUrl);

the app.update.url is read from the default branch, so we'll need to set the new pref on the default branch too for this value to be used

@@ +67,4 @@
>      return false;
>    }
>  
>    // Only update prefs if the app updates against mozilla.org servers

i think this part of the comment doesn't apply anymore either

@@ +71,2 @@
>    try {
>      let updateURL = Services.prefs.getCharPref(UPDATE_URL_PREF);

and read from the default branch

Comment 12

[Jake Maul [:jakem]]

Just updating from release-drivers@... we got the new Thawte certificate, and looking at its details we believe it will Just Work. We're setting it up and will test, then if all is well cut over to it.

If it does work well, we will not need the hotfix after all, and our concerns around TB/SM/Metro are all mooted. :)

Comment 13

[:Felipe Gomes (needinfo for replies!)]

(In reply to :Felipe Gomes from comment #11)
> > +    Services.prefs.setCharPref(UPDATE_CERTS_ISSUERNAME_PREF, CERT_ISSUERNAME);
> > +
> > +    let updateUrl = Services.prefs.getCharPref(UPDATE_URL_PREF);
> > +    updateUrl = UPDATE_URL_NEW_HOST + updateUrl.substr(UPDATE_URL_OLD_HOST.length);
> > +    Services.prefs.setCharPref(UPDATE_URL_PREF, updateUrl);
> 
> the app.update.url is read from the default branch, so we'll need to set the
> new pref on the default branch too for this value to be used

I did some testing and it turns out that it's not possible to set prefs on the default pref branch (they can be set during runtime but they're not saved after a restart). So we can't change the app.update.url easily through the hotfix.

On IRC rstrong said that we should leave this part out for now, and if needed, the hotfix can be used to add the new entries for app.update.certs.3.*.

Comment 14

[Blair McBride [:Unfocused] (UNAVAILABLE)]

(In reply to :Felipe Gomes from comment #13)
> I did some testing and it turns out that it's not possible to set prefs on
> the default pref branch (they can be set during runtime but they're not
> saved after a restart). So we can't change the app.update.url easily through
> the hotfix.

Ugh, yea, forgot that nsUpdateService only looks at the default pref. As a fallback, we could set it on the default branch and only uninstall the hotfix after the app is updated. 

But, looks a lot more likely that we won't be needing the hotfix anyway - so yay!

Comment 15

[:Gavin Sharp [email: [e-mail]]]

WONTFIX now, right?

Thanks for jumping on this quickly, Blair/Matt, and for the review help Felipe.

Comment 16

[Jake Maul [:jakem]]

Agreed... WONTFIX seems appropriate now. Thank you very much everyone!