1889299 - switch to new authenticode cert issued 2024-04-02

Status: ASSIGNED

(Release Engineering :: Release Automation: Signing, task)

Comment 1

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1889299 - add new authenticode signing certificate (#979)

Comment 2

[BMO Github Automation]

Attachment: [mozilla-releng/adhoc-signing] Bug 1889299 - re-sign esr115 installer with new authenticode cert for testing purposes (#206)

Comment 3

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1889299 - enable authenticode_202404 signing formats for firefox/… (#981)

Comment 4

[Julien Cristau [:jcristau]]

Attachment: Bug 1889299 - switch windows signing to autograph_authenticode_202404 format. r=taskgraph-reviewers,releng-reviewers,bhearsum

This switches us away from the code signing certificate expiring in
June, to a new one valid until 2027.

Comment 5

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1889299 - fix typo in signingscript config (#982)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/70c61aaf0509
switch windows signing to autograph_authenticode_202404 format. r=taskgraph-reviewers,releng-reviewers,bhearsum

Comment 7

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/70c61aaf0509

Comment 8

[Cosmin Sabou [:CosminS]]

Backed out out of central as requested by bhearsum for causing Bug 1894138: https://hg.mozilla.org/mozilla-central/rev/b7a1a8a3af7f8cb576e533dc6537f14dab105c55

Comment 9

[[e-mail] (:bhearsum)]

I dug a bit more. It turns out the new certificate has a different intermediate that it's been issued by.

Here's what we have on the current/old cert:

Signer's certificate:
	------------------
	Signer #0:
		Subject: /C=US/ST=California/L=Mountain View/O=Mozilla Corporation/OU=Firefox Engineering Operations/CN=Mozilla Corporation
		Issuer : /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Assured ID Code Signing CA
		Serial : 0C1CD3EEA47EDDA7A032573B014D0AFD
		Certificate expiration date:
			notBefore : Apr  9 00:00:00 2021 GMT
			notAfter : Jun 19 23:59:59 2024 GMT

And from the new cert:

Signer's certificate:
	------------------
	Signer #0:
		Subject: /C=US/ST=California/L=San Francisco/O=Mozilla Corporation/OU=Firefox Engineering Operations/CN=Mozilla Corporation
		Issuer : /C=US/O=DigiCert, Inc./CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
		Serial : 0737B0D0DCDCAB8D78D2F40CB122F93F
		Certificate expiration date:
			notBefore : Apr  2 00:00:00 2024 GMT
			notAfter : Jun 18 23:59:59 2027 GMT

Message digest algorithm: SHA256

This is problematic for the stub installer as well as the maintenance service.

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

Comment 10

[[e-mail] (:bhearsum)]

After a bit more examination, it appears that we did not actually break stub installs here due to the fact that the verification is run against one of the signed plugins, not against the full installer itself.

This means that unless we resign that plugin and land it at the same time, that we can't update the issuer pin in the stub.

Comment 11

[Julien Cristau [:jcristau]]

Requesting tracking because this needs to land for the 127 / 115.12 releases.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #10)

After a bit more examination, it appears that we did not actually break stub installs here due to the fact that the verification is run against one of the signed plugins, not against the full installer itself.

This means that unless we resign that plugin and land it at the same time, that we can't update the issuer pin in the stub.

This, as it turns out, is also wrong. download.exe is the full installer that we download.

When I tested last night I didn't see any issues with stub installs though, which is a bit confusing - I would've expected those to fail with a certificate check error.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #12)

(In reply to [e-mail] (:bhearsum) from comment #10)

After a bit more examination, it appears that we did not actually break stub installs here due to the fact that the verification is run against one of the signed plugins, not against the full installer itself.

This means that unless we resign that plugin and land it at the same time, that we can't update the issuer pin in the stub.

This, as it turns out, is also wrong. download.exe is the full installer that we download.

When I tested last night I didn't see any issues with stub installs though, which is a bit confusing - I would've expected those to fail with a certificate check error.

Ah - this was just a fluke of timing. We didn't publish the builds with the new cert until ~8pm eastern yesterday, and my tests were done about 30min prior to that.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #9)

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

The other "fun" implication of all of this is that unless I'm mistaken, old stub installers will fail to install the latest nightlies. I can think of two ways to deal with this:

• Go back to DigiCert and see if we can get a certificate from the previous intermediate.
• Keep the existing Bouncer entry for the full installer pointing at an installer that's signed with the current certificate, create a new one for the latest installers, and change the bouncer URL in all the necessary places (stub installer, bedrock, various tests and automation, etc.). This would result in out of date installs for users using an old stub installer as well, so it's got major downsides.

Nick - I'd appreciate your perspective on how much we ought to be doing here to avoid busting installs done from old stub installers.

Comment 15

[Julien Cristau [:jcristau]]

Bug 1079858 is the last time the intermediate changed and we had to deal with something similar.

Comment 16

[Nick Alexander :nalexander [he/him]]

(In reply to [e-mail] (:bhearsum) from comment #14)

(In reply to [e-mail] (:bhearsum) from comment #9)

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

The other "fun" implication of all of this is that unless I'm mistaken, old stub installers will fail to install the latest nightlies. I can think of two ways to deal with this:

• Go back to DigiCert and see if we can get a certificate from the previous intermediate.
• Keep the existing Bouncer entry for the full installer pointing at an installer that's signed with the current certificate, create a new one for the latest installers, and change the bouncer URL in all the necessary places (stub installer, bedrock, various tests and automation, etc.). This would result in out of date installs for users using an old stub installer as well, so it's got major downsides.

Nick - I'd appreciate your perspective on how much we ought to be doing here to avoid busting installs done from old stub installers.

Oh dear. So, we believe that there are significant numbers of old stub installers floating around, especially from parts of the unattributed funnel; but we can't (easily) identify them because adding to the stub installer telemetry is hard. The ticket to add the stub's version, which is what we need to understand this impact, never got done: https://bugzilla.mozilla.org/show_bug.cgi?id=1811167.

It's also our belief that installing an old version of Firefox (generally with a full installer, not a stub installer) contributes to poor Firefox user retention. Again, we don't know how frequently that experience occurs (but we should have better estimates).

I expect that it's well worth the effort to ask for a cert signed with the same intermediate to avoid this pain, but NI to Romain to see if he can say more here.

If we do have a breaking change, then at least we'll be able to test some theories around old installers :/

In the meantime, could we get a screenshot walkthrough of what the user experience with a certificate that doesn't validate looks like? If it redirects to mozilla.org in some way that's not a terrible outcome for us.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #16)

(In reply to [e-mail] (:bhearsum) from comment #14)

(In reply to [e-mail] (:bhearsum) from comment #9)

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

The other "fun" implication of all of this is that unless I'm mistaken, old stub installers will fail to install the latest nightlies. I can think of two ways to deal with this:

• Go back to DigiCert and see if we can get a certificate from the previous intermediate.
• Keep the existing Bouncer entry for the full installer pointing at an installer that's signed with the current certificate, create a new one for the latest installers, and change the bouncer URL in all the necessary places (stub installer, bedrock, various tests and automation, etc.). This would result in out of date installs for users using an old stub installer as well, so it's got major downsides.

Nick - I'd appreciate your perspective on how much we ought to be doing here to avoid busting installs done from old stub installers.

Oh dear. So, we believe that there are significant numbers of old stub installers floating around, especially from parts of the unattributed funnel; but we can't (easily) identify them because adding to the stub installer telemetry is hard. The ticket to add the stub's version, which is what we need to understand this impact, never got done: https://bugzilla.mozilla.org/show_bug.cgi?id=1811167.

It's also our belief that installing an old version of Firefox (generally with a full installer, not a stub installer) contributes to poor Firefox user retention. Again, we don't know how frequently that experience occurs (but we should have better estimates).

I expect that it's well worth the effort to ask for a cert signed with the same intermediate to avoid this pain, but NI to Romain to see if he can say more here.

If we do have a breaking change, then at least we'll be able to test some theories around old installers :/

In the meantime, could we get a screenshot walkthrough of what the user experience with a certificate that doesn't validate looks like? If it redirects to mozilla.org in some way that's not a terrible outcome for us.

Thank you for the perspective! I wasn't aware of the poor retention impacts of installing an older version, that's unfortunate :(.

Things moved very quickly here, I believe that Romain and jcristau synced up earlier today. https://docs.google.com/document/d/1Xzlp4wUNjxZPkOrvUq-yRYhDzhh8sMUCxokklqARJNI/edit#heading=h.j8rkaps9bkzi captures all the latest here.

The tl;dr version for this bug is that:

• We absolutely cannot get a certificate that is compatible with the current pins (we explored every angle of this with DigiCert)
• We have ideas about how we can avoid breaking old stub installers entirely (see the doc)

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c35b87066434
switch windows signing to autograph_authenticode_202404 format. r=taskgraph-reviewers,releng-reviewers,bhearsum,application-update-reviewers,bytesized

Comment 19

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/c35b87066434

Comment 20

[Cosmin Sabou [:CosminS]]

Backed out from central for causing bug 1896944: https://hg.mozilla.org/mozilla-central/rev/b7871b7f2c05e4076ad55ae4aa929b9862d4c8af