912675 - want to add an app.update.certs.3.* for digicert

Status: RESOLVED DUPLICATE of bug 921045

(Firefox :: Settings UI, defect)

Description

[Jake Maul [:jakem]]

Greetings,

The current Firefox is pinned to accept only either Equifax or Thawte for SSL certificates on aus3.mozilla.org (for updating). Equifax is in use right now, and Thawte is a backup.

This cert is about to expire and we'll have to renew, but we don't yet know which of those 2 vendors we can renew with. So, we don't want to change either of them at this point.

We do, however, want to add Digicert as an allowed vendor. They are our new/current SSL certificate vendor.

The new entries would be:

app.update.certs.3.commonName
aus3.mozilla.org

app.update.certs.3.issuerName
C=US; O=DigiCert Inc; CN=DigiCert Secure Server CA

We'd like to do this as soon as possible (uplift to Aurora/Beta even, so it goes out on next release)... the sooner it's available on release, the sooner it becomes realistically feasible to start using Digicert (we have to wait for a significant level of penetration, so we can't go right away... an extra 2-3 versions would help).

Comment 1

[Jake Maul [:jakem]]

Actually it looks like the current 2 specify the issuerName in the reverse order, and they use commas instead of semicolons... I don't know if it matters (probably does), but to match the other two it would actually look like this:

CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US

Comment 2

[Alex Keybl [:akeybl]]

I think this is the right bug for you

Comment 3

[Robert Strong (they/them - no direct email)]

Jake, since there is some confusion about the cert attribute values can you set this up on a system (or perhaps some other way) so I can verify it?

Comment 4

[Jake Maul [:jakem]]

You can see a comparable cert in action at pastebin.mozilla.org. That's identical to what the cert would be for aus3 (except obviously the CommonName is wrong and would be "aus3.mozilla.org"... but the issuer is correct). Will that suffice?

Comment 5

[u279076]

What sort of QA attention is needed here? Is it just a matter of us keeping a close eye on updates across branches?

Comment 6

[Jake Maul [:jakem]]

Yes, I believe that's sufficient. We are not intending to use this functionality any time soon, so effectively this should be a no-op for a few months... but the sooner we get it into the product, the sooner we can rely on it in the future, hence the request now rather than later.

Comment 7

[Jake Maul [:jakem]]

This situation we're in is starting to feel urgent. I have placed an order with Thawte, but their processing is slow and I cannot yet guarantee that the CA identification string will be an exact match for what Firefox has pinned. I have not yet found *any* way to order an Equifax certificate anymore... neither online or in our records. I don't think it's possible, but I'm still digging.

This leads me to a few questions:

1) How does the CA pinning check work? Will it look at only the intermediate that signed the aus3.mozilla.org certificate, or will it match if any cert in the chain has the right CA identifier? I'm afraid (but cannot yet confirm) that there might be an intermediate in place for Thawte that may not be the one called out in Firefox. If there is, and Firefox only checks the intermediate, and Equifax also falls through....

2) How quickly could we possibly spin up an addon hotfix to add Digicert? This seems a poor solution, but I'm starting to feel like we should really do it pronto, just in case. This at least gives us a worst-case-scenario road forward for Firefox 10+. Installations older than that seem likely to be either broken (can't update, so aus3 not working is irrelevant) or user-commanded not to update, so this might not be completely infeasible. Highly undesirable, but better than nothing if things go poorly.

2a) How are addon hotfixes deployed? Do they rely on aus3.mozilla.org in any way? I think not, documentation seems to indicate it relies on versioncheck.addons.mozilla.org instead. That would be good.

Comment 8

[Alex Keybl [:akeybl]]

Filed bug 913918

Comment 9

[Hector Zhao [:hectorz]]

(In reply to Jake Maul [:jakem] from comment #4)
> You can see a comparable cert in action at pastebin.mozilla.org. That's
> identical to what the cert would be for aus3 (except obviously the
> CommonName is wrong and would be "aus3.mozilla.org"... but the issuer is
> correct). Will that suffice?

issuerName of server cert at pastebin.mozilla.org: CN=DigiCert Secure Server CA,O=DigiCert Inc,C=US

Comment 10

[Hector Zhao [:hectorz]]

(In reply to Jake Maul [:jakem] from comment #7)
> 
> 1) How does the CA pinning check work? Will it look at only the intermediate
> that signed the aus3.mozilla.org certificate, or will it match if any cert
> in the chain has the right CA identifier? I'm afraid (but cannot yet
> confirm) that there might be an intermediate in place for Thawte that may
> not be the one called out in Firefox. If there is, and Firefox only checks
> the intermediate, and Equifax also falls through....

http://mxr.mozilla.org/mozilla-central/source/toolkit/modules/CertUtils.jsm#74

Looks like only the issuer information in the server cert is checked, it doesn't follow the chains.

Comment 12

[Blair McBride [:Unfocused] (UNAVAILABLE)]

(In reply to Jake Maul [:jakem] from comment #7)
> 2a) How are addon hotfixes deployed? Do they rely on aus3.mozilla.org in any
> way? I think not, documentation seems to indicate it relies on
> versioncheck.addons.mozilla.org instead. That would be good.

It's done entirely through AMO - just like a normal addon update. So it'll work fine.

Comment 13

[Jake Maul [:jakem]]

Thanks Blair, that helps a lot... it means we have a road forward for Firefox 10+ users even in the worst case scenario.


From release-drivers@, it appears that Thunderbird, Seamonkey, and Metro Firefox are likely affected by this as well, and they don't support addon hotfixing. So they'll need some other kind of attention.

Presuming that is the case (they are pinned, and can't be addon hotfixed), I only see 2 possible roads ahead for them:


1) There's still the chance that the on-order Thawte cert will match the pin in app.update.certs. In that case, hooray, no immediate action needed.

2) If the on-order Thawte cert doesn't match, these will likely need a new release very very quickly (to allow Digicert and move users to "aus3-1.mozilla.org" - discussed in the release-drivers@ thread), in order to get reasonable penetration before the existing cert expires. Any users not updated in time will be directed to a full installer fallback, and they'll have to do a pave-over install.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Jake Maul [:jakem] from comment #13)
> From release-drivers@, it appears that Thunderbird, Seamonkey, and Metro
> Firefox are likely affected by this as well, and they don't support addon
> hotfixing. So they'll need some other kind of attention.

Thunderbird very likely is. It has the same prefs set, but I'm not sure how we'd go about testing in the wild.

SeaMonkey won't be as it uses aus2-community.mozilla.org. 

I'm sceptical of Metro Firefox being affected because we only have Metro enabled in Nightly builds, and have recently been talking about when it would advance further. We also haven't been asked to ensure updates work beyond Nightly yet. Blair, could you point to a bug for what you meant about Firefox 24 & Metro ?

Comment 15

[Jake Maul [:jakem]]

Now that we're out of the woods on this, I suspect we'll want to re-evaluate this altogether. There's a post-mortem on Thursday to discuss, so let's not do anything rash before then. :)

We might end up WONTFIXing this bug and opening a new, clearer one, with a new direction. Notably, I expect we'll want to remove Equifax, move Thawte to the #1 spot (:rstrong indicated to me that Firefox may throw a console warning if the cert in use isn't the first one), and make Digicert #2. We may also want to consider adding a #3 (TBD, possibly Verisign), just to give us more options in the future.

We also want to discuss revamping this design altogether, or at least adding some regular processes to it. These settings depend on 3rd party vendors, so at the very least we need to have some form of regular review to make sure we'll be able to cope with changes those vendors might make.

Comment 16

[Robert Strong (they/them - no direct email)]

I've added the new certificate in bug 921045 so duping.

(In reply to Jake Maul [:jakem] from comment #15)
>...
> We also want to discuss revamping this design altogether, or at least adding
> some regular processes to it. These settings depend on 3rd party vendors, so
> at the very least we need to have some form of regular review to make sure
> we'll be able to cope with changes those vendors might make.
I have already removed the checks on Windows in bug 928489 since we verify the signature on the mar file itself on Windows and the checks will be removed on Mac and Linux after mar signing is implemented for those platforms in bug 920750.