1894689 - update cert issuer name in windows maintenance service

Status: RESOLVED FIXED

(Firefox :: Installer, task, P1)

Comment 1

[Julien Cristau [:jcristau]]

Attachment: Bug 1894689 - update pinned cert issuers in windows maintenance service. r?bhearsum!,#application-update-reviewers!

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c8f1b6d6898
update pinned cert issuers in windows maintenance service. r=bhearsum,application-update-reviewers,bytesized

Comment 3

[Julien Cristau [:jcristau]]

Attachment: Bug 1894689 - update pinned cert issuers in windows maintenance service. r?bhearsum!,#application-update-reviewers!

Original Revision: https://phabricator.services.mozilla.com/D209245

Comment 4

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: maintenance service can't update after we update the signing cert
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: tbd
• Risk associated with taking this patch: medium
• Explanation of risk level: The patch is simple enough, but as with anything that touches updates, bad consequences if we get it wrong
• String changes made/needed: n/a
• Is Android affected?: no

Comment 5

[Julien Cristau [:jcristau]]

Attachment: Bug 1894689 - update pinned cert issuers in windows maintenance service. r?bhearsum!,#application-update-reviewers!

Original Revision: https://phabricator.services.mozilla.com/D209245

Comment 6

[Phabricator Automation]

esr115 Uplift Approval Request

• User impact if declined: maintenance service can't update after we update the signing cert
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: tbd
• Risk associated with taking this patch: medium
• Explanation of risk level: The patch is simple enough, but as with anything that touches updates, bad consequences if we get it wrong
• String changes made/needed: n/a
• Is Android affected?: no

Comment 7

[Julien Cristau [:jcristau]]

On each channel, we're going to need a watershed on a release with this patch before we can roll out the new signing certificate, so ideally this can go out to 126 / 115.11, and the new cert in 127 / 115.12.

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9c8f1b6d6898

Comment 9

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

I have some questions about this. First: are we 100% confident that updates always update the maintenance service? I hope that's true but I don't know that's true, and I vaguely remember there were issues with this -- but they might have been automation only, i.e., situations where automation didn't have the latest MMS.

Second, the maintenance service is a component shared across Firefox installations. Is it possible that an older installation might be using a newer MMS and try to install an older MAR? I.e., do we need to accept both certificates/issuers in order to service multiple installations, and not only trust a watershed to handle each individual installation?

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #9)

I have some questions about this. First: are we 100% confident that updates always update the maintenance service? I hope that's true but I don't know that's true, and I vaguely remember there were issues with this -- but they might have been automation only, i.e., situations where automation didn't have the latest MMS.

I don't think I can provide any more confidence than you can that updates in the real world will update the registry keys.

We definitely have issues with this in automation though, see https://bugzilla.mozilla.org/show_bug.cgi?id=1692074 for the longstanding bug.

Second, the maintenance service is a component shared across Firefox installations. Is it possible that an older installation might be using a newer MMS and try to install an older MAR? I.e., do we need to accept both certificates/issuers in order to service multiple installations, and not only trust a watershed to handle each individual installation?

Good callout - we hadn't thought about this scenario until now.

Just to make sure I understand: I believe what you're getting at is that we could have installs X & Y, and have a sequence of events like this:

• X updates to a build containing the new issuer values and updates the registry keys.
• X updates to a new build with a new maintenance service signed by the new authenticode cert with the just-added issuer value successfully through the maintenance service
• Y somehow gets an update to a build with the old issuer values (perhaps this build is behind the last watershed), and updates the registry keys
• X tries to update again to a new build signed by the new authenticode cert, and the maintenance service fails to verify it due to Y overwriting the issuer fails

If I understand this correctly, then yes - I believe this could happen. In scenarios where X can elevate it can probably be worked around - but obviously background updates are the ideal.

I imagine we could mitigate this by using new registry keys for the new issuers, to avoid any sort of collisions?

Comment 11

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to [e-mail] (:bhearsum) from comment #10)

(In reply to Nick Alexander :nalexander [he/him] from comment #9)

I have some questions about this. First: are we 100% confident that updates always update the maintenance service? I hope that's true but I don't know that's true, and I vaguely remember there were issues with this -- but they might have been automation only, i.e., situations where automation didn't have the latest MMS.

I don't think I can provide any more confidence than you can that updates in the real world will update the registry keys.

We definitely have issues with this in automation though, see https://bugzilla.mozilla.org/show_bug.cgi?id=1692074 for the longstanding bug.

Yes, this was what I was thinking of. Based on the discussion in the ticket you link, I'm confident we're updating the MMS if necessary at install time. See also https://searchfox.org/mozilla-central/rev/38377227b8f96fda8f418db614e6a8aa67d01c31/browser/installer/windows/nsis/shared.nsh#138-156 in PostUpdate.

Second, the maintenance service is a component shared across Firefox installations. Is it possible that an older installation might be using a newer MMS and try to install an older MAR? I.e., do we need to accept both certificates/issuers in order to service multiple installations, and not only trust a watershed to handle each individual installation?

Good callout - we hadn't thought about this scenario until now.

Just to make sure I understand: I believe what you're getting at is that we could have installs X & Y, and have a sequence of events like this:

• X updates to a build containing the new issuer values and updates the registry keys.
• X updates to a new build with a new maintenance service signed by the new authenticode cert with the just-added issuer value successfully through the maintenance service
• Y somehow gets an update to a build with the old issuer values (perhaps this build is behind the last watershed), and updates the registry keys

I was actually thinking that installation Y could get an old MAR, with the old issuer values, and then the updated MMS would reject those MARs. I think this must happen: old MARs, to get X to the watershed, must be signed with the old issuer values (or they can't apply). Then the first to get past the watershed will change the expected issuer, blocking all of the other installations from applying MARs to the watershed.

Perhaps this isn't an issue: fallbacks will fail, and eventually a complete MAR will be downloaded, and that complete MAR will use the new values? I think a diagram is needed here, 'cuz it's not clear what MARs will have what values.

• X tries to update again to a new build signed by the new authenticode cert, and the maintenance service fails to verify it due to Y overwriting the issuer fails

If I understand this correctly, then yes - I believe this could happen. In scenarios where X can elevate it can probably be worked around - but obviously background updates are the ideal.

I imagine we could mitigate this by using new registry keys for the new issuers, to avoid any sort of collisions?

Possibly? I don't really understand the details here. I'm aware time is short but it'll have to wait for Monday for me.

Comment 12

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: maintenance service can't update after we update the signing cert
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: tbd
• Risk associated with taking this patch: medium
• Explanation of risk level: The patch is simple enough, but as with anything that touches updates, bad consequences if we get it wrong
• String changes made/needed: n/a
• Is Android affected?: no

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #11)
Let's take this part first to try to clear things up...

Perhaps this isn't an issue: fallbacks will fail, and eventually a complete MAR will be downloaded, and that complete MAR will use the new values? I think a diagram is needed here, 'cuz it's not clear what MARs will have what values.

I'm happy to diagram this out if needed, but AIUI it's fairly simple:

• MARs prior to the watershed watershed will have their maintenance service signed with the old cert, and the registry values will contain the current cert data in CERTIFICATE_ISSUER and the ancient cert data in CERTIFICATE_ISSUER_PREVIOUS.
• MARs at that watershed will have their maintenance service signed with the old cert, and the registry values will contain the new data in CERTIFICATE_ISSUER and the current cert data in CERTIFICATE_ISSUER_PREVIOUSL
• MARs after that will have their maintenance service signed with the new cert, and the registry values will contain the new data in CERTIFICATE_ISSUER and the current cert data in CERTIFICATE_ISSUER_PREVIOUSL

Also, Julien noted to me that the registry locations are are per-install, to perhaps there are fewer scenarios here to consider than I first thought...

Second, the maintenance service is a component shared across Firefox installations. Is it possible that an older installation might be using a newer MMS and try to install an older MAR? I.e., do we need to accept both certificates/issuers in order to service multiple installations, and not only trust a watershed to handle each individual installation?

Good callout - we hadn't thought about this scenario until now.

Just to make sure I understand: I believe what you're getting at is that we could have installs X & Y, and have a sequence of events like this:

• X updates to a build containing the new issuer values and updates the registry keys.
• X updates to a new build with a new maintenance service signed by the new authenticode cert with the just-added issuer value successfully through the maintenance service
• Y somehow gets an update to a build with the old issuer values (perhaps this build is behind the last watershed), and updates the registry keys

I was actually thinking that installation Y could get an old MAR, with the old issuer values, and then the updated MMS would reject those MARs. I think this must happen: old MARs, to get X to the watershed, must be signed with the old issuer values (or they can't apply). Then the first to get past the watershed will change the expected issuer, blocking all of the other installations from applying MARs to the watershed.

I think we're saying the same thing here? More generally: you can apply MARs up to and including the watershed with either the old or new registry keys. (Because we have the new and current cert in CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS respectively.) MARs from after the watershed require the new registry keys (because the maintenance service contained within them are signed by the new cert).

However, given the per-install registry keys, maybe this is not relevant?

Comment 14

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to [e-mail] (:bhearsum) from comment #13)

(In reply to Nick Alexander :nalexander [he/him] from comment #11)
Let's take this part first to try to clear things up...

Perhaps this isn't an issue: fallbacks will fail, and eventually a complete MAR will be downloaded, and that complete MAR will use the new values? I think a diagram is needed here, 'cuz it's not clear what MARs will have what values.

I'm happy to diagram this out if needed, but AIUI it's fairly simple:

• MARs prior to the watershed watershed will have their maintenance service signed with the old cert, and the registry values will contain the current cert data in CERTIFICATE_ISSUER and the ancient cert data in CERTIFICATE_ISSUER_PREVIOUS.
• MARs at that watershed will have their maintenance service signed with the old cert, and the registry values will contain the new data in CERTIFICATE_ISSUER and the current cert data in CERTIFICATE_ISSUER_PREVIOUSL
• MARs after that will have their maintenance service signed with the new cert, and the registry values will contain the new data in CERTIFICATE_ISSUER and the current cert data in CERTIFICATE_ISSUER_PREVIOUSL

Okay, I see what you mean, I hadn't fully worked through the previous value affordance. This makes sense to me.

Also, Julien noted to me that the registry locations are are per-install, to perhaps there are fewer scenarios here to consider than I first thought...

TIL; I didn't realize we had per-installation registry locations.

Second, the maintenance service is a component shared across Firefox installations. Is it possible that an older installation might be using a newer MMS and try to install an older MAR? I.e., do we need to accept both certificates/issuers in order to service multiple installations, and not only trust a watershed to handle each individual installation?

Good callout - we hadn't thought about this scenario until now.

Just to make sure I understand: I believe what you're getting at is that we could have installs X & Y, and have a sequence of events like this:

• X updates to a build containing the new issuer values and updates the registry keys.
• X updates to a new build with a new maintenance service signed by the new authenticode cert with the just-added issuer value successfully through the maintenance service
• Y somehow gets an update to a build with the old issuer values (perhaps this build is behind the last watershed), and updates the registry keys

I was actually thinking that installation Y could get an old MAR, with the old issuer values, and then the updated MMS would reject those MARs. I think this must happen: old MARs, to get X to the watershed, must be signed with the old issuer values (or they can't apply). Then the first to get past the watershed will change the expected issuer, blocking all of the other installations from applying MARs to the watershed.

I think we're saying the same thing here? More generally: you can apply MARs up to and including the watershed with either the old or new registry keys. (Because we have the new and current cert in CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS respectively.) MARs from after the watershed require the new registry keys (because the maintenance service contained within them are signed by the new cert).

However, given the per-install registry keys, maybe this is not relevant?

I concur.

I'm satisfied that the issue I thought might arise with multiple installations not having the current issuers declared should not arise in the wild. Thanks, Ben!

Comment 15

[Ada C.]

Verified that build updates to the latest Nightly 127 build is work correctly. Testing was done on Windows 10 and Windows 11, using different installer builds (.msi, .zip, .exe, .msix ), starting from builds before the fix and builds with the fix. - detailed test results here -> https://docs.google.com/spreadsheets/d/1cn38geXHsRDd1N0Gzp3itFTRaeNcyiQ3KkCAOxFZLTc/edit#gid=142550933

Comment 16

[Donal Meehan [:dmeehan]]

Comment on attachment 9399934 [details]
Bug 1894689 - update pinned cert issuers in windows maintenance service. r?bhearsum!,#application-update-reviewers!

Rejecting release uplift request.
This will ride the train to release with Fx127 since Bug 1896944 is also required.

Comment 17

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr115/rev/2950bf83466c

Comment 18

[Ada C.]

Verified that build updates to the latest Firefox ESR build work correctly. Testing was done on Windows 7, Windows 10 and Windows 11, using different installer builds (.msi, .zip, .exe, .msix ) - detailed test results here -> https://docs.google.com/spreadsheets/d/1cn38geXHsRDd1N0Gzp3itFTRaeNcyiQ3KkCAOxFZLTc/edit#gid=332700012