Closed Bug 1768253 Opened 2 years ago Closed 2 years ago

Renew aus5.mozilla.org 2022

Categories

(Cloud Services :: Operations: Balrog, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jbuck, Assigned: jbuck)

References

Details

Attachments

(1 file)

On 2022-07-04 the current certificate for aus5.mozilla.org is expiring. The certificate is currently a Digicert-issued certificate. If possible, I'd like to switch this certificate to a Let's Encrypt-issued certificate. Assuming it is, I'd like to plan out a time for testing the Let's Encrypt certificate as well as an actual rotation period ahead of the early July holidays.

Here's a link to the current compatibility matrix for Let's Encrypt certificates: https://letsencrypt.org/docs/certificate-compatibility/ . Would any of these incompatible platforms prevent us from using Let's Encrypt?

I'm not sure how soon after bug 1760527 rolls out to release (and esr) we're ok with breaking GMP updates for earlier versions.

Per https://mozilla-balrog.readthedocs.io/en/latest/client_domains.html#pinning-requirements, thunderbird <51 also pins to digicert, so we'll want to give them a heads-up, but I don't think that'll block.

On Slack we collectively recalled that the reason we wanted to remove GMP certificate pinning entirely was that the Digicert intermediate CA "old" DigiCert SHA2 Secure Server CA was expiring on 2023-03-08 which means that it can't be used to issue certificates anymore. :bhearsum pointed out that GMP certificate pinning is done by matching the CN, not the certificate serial number or sha256 hash, so we might be able to request a new certificate for balrog using the "new" DigiCert SHA2 Secure Server CA intermediate CA.

:johnb - Hi, could you file a support ticket with Digicert asking what the procedure is to request that a certificate be issued with a different intermediate CA? We'd like to renew aus5.mozilla.org (and some other related domains, I will generate a CSR) using the "DigiCert SHA2 Secure Server CA" intermediate instead of the default "DigiCert TLS RSA SHA256 2020 CA1" intermediate. The reason we need this alternate intermediate is that Firefox <= 100 is pinned to the intermediate CN CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US.

Flags: needinfo?(jbircher)
Summary: Renew aus5.mozilla.org - switch to Let's Encrypt → Renew aus5.mozilla.org 2022

:johnb granted me access to Digicert (ty!) and I've sent an email to Digicert support requesting:

  1. Confirmation we can get a certificate issued by DigiCert SHA2 Secure Server CA
  2. How I actually renew the certificate using the alternate intermediate CA

I will update this bug with the response from Digicert.

Flags: needinfo?(jbircher)

Digicert support says it's possible and we need to have our Digicert account manager make a change - I've sent an email to them asking what the process is.

Our account manager made the change and I've submitted the renewal order

(In reply to Julien Cristau [:jcristau] from comment #1)

I'm not sure how soon after bug 1760527 rolls out to release (and esr) we're ok with breaking GMP updates for earlier versions.

Per https://mozilla-balrog.readthedocs.io/en/latest/client_domains.html#pinning-requirements, thunderbird <51 also pins to digicert, so we'll want to give them a heads-up, but I don't think that'll block.

FYI the pref flip for this is rolling out in 101 and will not be back ported. We could enable content sig back to 95 but would like to avoid that. Cert pinning is used in initial installs of Widevine and OpenH264, so we would prefer to not break that for older versions.

(In reply to Jon Buckley [:jbuck] from comment #2)

On Slack we collectively recalled that the reason we wanted to remove GMP certificate pinning entirely was that the Digicert intermediate CA "old" DigiCert SHA2 Secure Server CA was expiring on 2023-03-08 which means that it can't be used to issue certificates anymore. :bhearsum pointed out that GMP certificate pinning is done by matching the CN, not the certificate serial number or sha256 hash, so we might be able to request a new certificate for balrog using the "new" DigiCert SHA2 Secure Server CA intermediate CA.

I'm curious about this, does this mean the current cert pinning code gmp install manager is using will continue to work through 2023? If so, we are good here. But if cert pinning starts to fail in July we may get some complaints from old Firefox / ESR users. Obviously our answer there would be to update to Fx 101 and ESR 102.0. However 102 will have only been out for a few weeks (shipping 2022-06-28).

If the answer here we're not sure if would break I'd appreciate some guidance on how QA can test the scenario of the cert expiring.

(In reply to Jim Mathies [:jimm] from comment #7)

I'm curious about this, does this mean the current cert pinning code gmp install manager is using will continue to work through 2023? If so, we are good here. But if cert pinning starts to fail in July we may get some complaints from old Firefox / ESR users. Obviously our answer there would be to update to Fx 101 and ESR 102.0. However 102 will have only been out for a few weeks (shipping 2022-06-28).

If the answer here we're not sure if would break I'd appreciate some guidance on how QA can test the scenario of the cert expiring.

Yes, we believe that issuing the aus5.mozilla.org certificate using DigiCert SHA2 Secure Server CA will allow GMP certificate pinning to continue to work as-is. I am currently working with Digicert to get that certificate renewed now.

If you'd like, I can spin up some additional load balancers using the renewed aus5 certificate and/or a Let's Encrypt certificate for QA for testing.

(In reply to Jon Buckley [:jbuck] from comment #8)

(In reply to Jim Mathies [:jimm] from comment #7)

I'm curious about this, does this mean the current cert pinning code gmp install manager is using will continue to work through 2023? If so, we are good here. But if cert pinning starts to fail in July we may get some complaints from old Firefox / ESR users. Obviously our answer there would be to update to Fx 101 and ESR 102.0. However 102 will have only been out for a few weeks (shipping 2022-06-28).

If the answer here we're not sure if would break I'd appreciate some guidance on how QA can test the scenario of the cert expiring.

Yes, we believe that issuing the aus5.mozilla.org certificate using DigiCert SHA2 Secure Server CA will allow GMP certificate pinning to continue to work as-is. I am currently working with Digicert to get that certificate renewed now.

I'll just pop in to +1 this -- there's no reason to think that this should break anything. (We've done cert renewals like this before without issue.)

Okay, the new certificate has been issued, and I can rotate it in at any time. I can also rotate back to the current certificate if we run into issues for the next 30 days or so.

Does anyone have a date/time preference this week?

Certificate rotation complete, please let me know if you see anything weird

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED

Thanks Jon!

Assignee: nobody → jbuckley

Embarrassingly I didn't update our infra code to point at the new certificate so when a new version of Balrog was deployed to switched back to the old certificate! I manually switched back to the new certificate, and here's a PR to fix that for good: https://github.com/mozilla-services/cloudops-infra/pull/4145

See Also: → 1772799
Blocks: 1834817
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: