1768253 - Renew aus5.mozilla.org 2022

Status: RESOLVED FIXED

(Cloud Services :: Operations: Balrog, task)

Description

[Jon Buckley [:jbuck]]

On 2022-07-04 the current certificate for aus5.mozilla.org is expiring. The certificate is currently a Digicert-issued certificate. If possible, I'd like to switch this certificate to a Let's Encrypt-issued certificate. Assuming it is, I'd like to plan out a time for testing the Let's Encrypt certificate as well as an actual rotation period ahead of the early July holidays.

Here's a link to the current compatibility matrix for Let's Encrypt certificates: https://letsencrypt.org/docs/certificate-compatibility/ . Would any of these incompatible platforms prevent us from using Let's Encrypt?

Comment 1

[Julien Cristau [:jcristau]]

I'm not sure how soon after bug 1760527 rolls out to release (and esr) we're ok with breaking GMP updates for earlier versions.

Per https://mozilla-balrog.readthedocs.io/en/latest/client_domains.html#pinning-requirements, thunderbird <51 also pins to digicert, so we'll want to give them a heads-up, but I don't think that'll block.

Comment 2

[Jon Buckley [:jbuck]]

On Slack we collectively recalled that the reason we wanted to remove GMP certificate pinning entirely was that the Digicert intermediate CA "old" DigiCert SHA2 Secure Server CA was expiring on 2023-03-08 which means that it can't be used to issue certificates anymore. :bhearsum pointed out that GMP certificate pinning is done by matching the CN, not the certificate serial number or sha256 hash, so we might be able to request a new certificate for balrog using the "new" DigiCert SHA2 Secure Server CA intermediate CA.

Comment 3

[Jon Buckley [:jbuck]]

:johnb - Hi, could you file a support ticket with Digicert asking what the procedure is to request that a certificate be issued with a different intermediate CA? We'd like to renew aus5.mozilla.org (and some other related domains, I will generate a CSR) using the "DigiCert SHA2 Secure Server CA" intermediate instead of the default "DigiCert TLS RSA SHA256 2020 CA1" intermediate. The reason we need this alternate intermediate is that Firefox <= 100 is pinned to the intermediate CN CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US.

Comment 4

[Jon Buckley [:jbuck]]

:johnb granted me access to Digicert (ty!) and I've sent an email to Digicert support requesting:

1. Confirmation we can get a certificate issued by DigiCert SHA2 Secure Server CA
2. How I actually renew the certificate using the alternate intermediate CA

I will update this bug with the response from Digicert.

Comment 5

[Jon Buckley [:jbuck]]

Digicert support says it's possible and we need to have our Digicert account manager make a change - I've sent an email to them asking what the process is.

Comment 6

[Jon Buckley [:jbuck]]

Our account manager made the change and I've submitted the renewal order

Comment 7

[Jim Mathies [:jimm]]

(In reply to Julien Cristau [:jcristau] from comment #1)

I'm not sure how soon after bug 1760527 rolls out to release (and esr) we're ok with breaking GMP updates for earlier versions.

Per https://mozilla-balrog.readthedocs.io/en/latest/client_domains.html#pinning-requirements, thunderbird <51 also pins to digicert, so we'll want to give them a heads-up, but I don't think that'll block.

FYI the pref flip for this is rolling out in 101 and will not be back ported. We could enable content sig back to 95 but would like to avoid that. Cert pinning is used in initial installs of Widevine and OpenH264, so we would prefer to not break that for older versions.

(In reply to Jon Buckley [:jbuck] from comment #2)

On Slack we collectively recalled that the reason we wanted to remove GMP certificate pinning entirely was that the Digicert intermediate CA "old" DigiCert SHA2 Secure Server CA was expiring on 2023-03-08 which means that it can't be used to issue certificates anymore. :bhearsum pointed out that GMP certificate pinning is done by matching the CN, not the certificate serial number or sha256 hash, so we might be able to request a new certificate for balrog using the "new" DigiCert SHA2 Secure Server CA intermediate CA.

I'm curious about this, does this mean the current cert pinning code gmp install manager is using will continue to work through 2023? If so, we are good here. But if cert pinning starts to fail in July we may get some complaints from old Firefox / ESR users. Obviously our answer there would be to update to Fx 101 and ESR 102.0. However 102 will have only been out for a few weeks (shipping 2022-06-28).

If the answer here we're not sure if would break I'd appreciate some guidance on how QA can test the scenario of the cert expiring.

Comment 8

[Jon Buckley [:jbuck]]

(In reply to Jim Mathies [:jimm] from comment #7)

I'm curious about this, does this mean the current cert pinning code gmp install manager is using will continue to work through 2023? If so, we are good here. But if cert pinning starts to fail in July we may get some complaints from old Firefox / ESR users. Obviously our answer there would be to update to Fx 101 and ESR 102.0. However 102 will have only been out for a few weeks (shipping 2022-06-28).

If the answer here we're not sure if would break I'd appreciate some guidance on how QA can test the scenario of the cert expiring.

Yes, we believe that issuing the aus5.mozilla.org certificate using DigiCert SHA2 Secure Server CA will allow GMP certificate pinning to continue to work as-is. I am currently working with Digicert to get that certificate renewed now.

If you'd like, I can spin up some additional load balancers using the renewed aus5 certificate and/or a Let's Encrypt certificate for QA for testing.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Jon Buckley [:jbuck] from comment #8)

(In reply to Jim Mathies [:jimm] from comment #7)

I'm curious about this, does this mean the current cert pinning code gmp install manager is using will continue to work through 2023? If so, we are good here. But if cert pinning starts to fail in July we may get some complaints from old Firefox / ESR users. Obviously our answer there would be to update to Fx 101 and ESR 102.0. However 102 will have only been out for a few weeks (shipping 2022-06-28).

If the answer here we're not sure if would break I'd appreciate some guidance on how QA can test the scenario of the cert expiring.

Yes, we believe that issuing the aus5.mozilla.org certificate using DigiCert SHA2 Secure Server CA will allow GMP certificate pinning to continue to work as-is. I am currently working with Digicert to get that certificate renewed now.

I'll just pop in to +1 this -- there's no reason to think that this should break anything. (We've done cert renewals like this before without issue.)

Comment 10

[Jon Buckley [:jbuck]]

Okay, the new certificate has been issued, and I can rotate it in at any time. I can also rotate back to the current certificate if we run into issues for the next 30 days or so.

Does anyone have a date/time preference this week?

Comment 11

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-releng/balrog/pull/2531

Comment 12

[Jon Buckley [:jbuck]]

Certificate rotation complete, please let me know if you see anything weird

Comment 13

[Julien Cristau [:jcristau]]

Thanks Jon!

Comment 14

[Jon Buckley [:jbuck]]

Embarrassingly I didn't update our infra code to point at the new certificate so when a new version of Balrog was deployed to switched back to the old certificate! I manually switched back to the new certificate, and here's a PR to fix that for good: https://github.com/mozilla-services/cloudops-infra/pull/4145