1549018 - create a transition plan to prepare for the add-on signing root expiring in 2025

Status: ASSIGNED

(Toolkit :: Add-ons Manager, task, P2)

Description

[Daniel Veditz [:dveditz]]

Inspired by bug 1548973, the add-on signing root is going to expire in 2025 so we need a transition plan ("Not After : Mar 14 22:53:57 2025 GMT"). Ideally we don't want to break existing signed add-ons so I think that means creating a new security/apps/addons-public.crt with the same issuer name and key.

Any old Firefox that has not gotten the updated cert will have all its add-ons fail so we need to transition well before the expiration date (couple of years?) so that long-lived ESRs and watershed releases have the future cert in time.

2021 or 2022 might be a good time to do this.

Comment 1

[Alexis Murzeau]

Hi,

What about using a addon timestamp so signatures and certificates can't expire even when never updated (because they maybe still work despite a not so available maintainer) ?
But the timestamp might have to be signed too then ...

I think this is what Windows does to check drivers signatures.

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Presumably we'll need to update the new intermediate at that time as well (it expires around the same time as the root now, I believe).
In any case, I think comment 0 outlines a good approach (unless we want to have a new key? not sure what best practices are about that (or will be at that time)).

Comment 3

[Ryan Hendrickson [rhendric on GitLab, GitHub]]

This is possibly a dumb question, but why does the signing key need to have an expiration date at all (or one more recent than, say, 2200)? If the plan is already to replace these keys every few years, than the usual motivation for having certificates expire (as I understand it: if malware manages to get a hold of a very old private key, you don't want them to be able to spoof your current system) doesn't apply to any browser that's been updated in the past few years. The main difference is that people powering on computers that haven't been used in a decade won't suddenly find that none of their add-ons work; instead, they might theoretically be vulnerable to malware-installed addons if the malware author has access to a private key from the appropriate date range. IMO, that's a reasonable trade-off.

Comment 4

[Daniel Veditz [:dveditz]]

Since this has come up a couple of times I should clarify that I'm not presupposing that signed add-ons will work the same way in the future. We could "transition" to something that doesn't require certificates at all, I suppose (signed .MAR files don't use certificates, for example). But we will still need a transition plan for how to handle old add-ons and old clients and how to get users on to the new system.

Comment 5

[Mike Hoye [:mhoye]]

Proposal: As a lightweight failsafe we can act on today, we could add a test to somewhere appropriate that checks the date and fails with an informative message if the date is later than, say, October 1st, 2024.

Comment 6

[Kris Maglione [:kmag]]

(In reply to Mike Hoye [:mhoye] from comment #5)

Proposal: As a lightweight failsafe we can act on today, we could add a test to somewhere appropriate that checks the date and fails with an informative message if the date is later than, say, October 1st, 2024.

Something along those lines is probably a good idea, but 2024 is really too late. There are more options on the table if we have at least a full ESR cycle of lead time.

Comment 7

[Julien Vehent]

I'm working on this with mgoodwin in Q4. Will have a proposal to present to this group by Berlin.

Comment 8

[Shane Caraveo (:mixedpuppy)]

Does the whiteboard markup mean that the work on this is scheduled?

Comment 9

[Mike Hoye [:mhoye]]

(In reply to Kris Maglione [:kmag] from comment #6)

Something along those lines is probably a good idea, but 2024 is really too late. There are more options on the table if we have at least a full ESR cycle of lead time.

If the general approach of putting a deadman switch into the build process to flag impending certificate expiration is acceptable, I can put one for, say Q2-2023?

Comment 10

[Shane Caraveo (:mixedpuppy)]

(In reply to Mike Hoye [:mhoye] from comment #9)

(In reply to Kris Maglione [:kmag] from comment #6)

Something along those lines is probably a good idea, but 2024 is really too late. There are more options on the table if we have at least a full ESR cycle of lead time.

If the general approach of putting a deadman switch into the build process to flag impending certificate expiration is acceptable, I can put one for, say Q2-2023?

I think that is a great idea, if this doesn't progress another 2 years, we have a problem.

Comment 11

[AJ Bahnken [:ajvb] (she/her)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #8)

Does the whiteboard markup mean that the work on this is scheduled?

No, it essentially means "Needs Triage in 2021". I'm bringing this one up in our next team meeting though.

Comment 12

[Rob Wu [:robwu]]

Attachment: Bug 1549018 - Use a fixed time for addon signature checks

The AddonsPublicRoot certificate (security/apps/addons-public.crt) has
a validity from 2015 until 2025. This is a timebomb that causes all
add-ons to be disabled in March 2025. To avoid armagadd-on-3.0, we
should ignore notBefore / notAfter dates of the root certificate.

There is existing logic to ignore validity errors of end entity
certificates. We cannot add a similar check for validity errors of the
issuer certificate, because that could result in shadowing other
lower-ranking but still serious validation errors.

To still achieve the same effect, use a fixed time between 2015 and 2025
instead of the current time in certificate validation checks.

Comment 13

[Rob Wu [:robwu]]

While looking into bug 1713628, I found a way to disable the timebomb on Firefox's side.

If Autograph is fine with signing XPIs using an expired root certificate, then the patch at https://phabricator.services.mozilla.com/D119812 would be an effective way to avoid armagadd-on-3.0.

I've verified that it works using

faketime 2026-01-01 ./objdir-debug/dist/bin/firefox --no-remote -profile /tmp/profdir

I've also simulated a renewed certificate by changing the hard-coded timestamp to a date in the far past (removing a digit). When I try to install an extension after doing that, in a debug build the assertion is triggered, which would allow us to detect incompatibilities between the implemented logic and the certificate that's in use:

Assertion failure: result != mozilla::pkix::Result::ERROR_NOT_YET_VALID_ISSUER_CERTIFICATE, at security/apps/AppSignatureVerification.cpp:645

Comment 14

[Rob Wu [:robwu]]

Kris, do you recall why the patch from bug 1549010 (https://phabricator.services.mozilla.com/D29928) got backed out indefinitely? To resolve armagadd-on-2.0 (bug 1548973) at the time a different approach was used (bug 1549061 followed by bug 1549249), but that is still not a permanent solution (hence this bug). The approach to use a fixed time for verification (i.e. bug 1549010 and my proposed patch here) would remove the time bomb and avoid a future armagadd-on-3.0 incident.

I'm aware that choosing a fixed date "in the past" can currently result in a failed signature verification because the end-entity certificate used to sign the add-on has a notBefore set to the time of signing (which would be a future date relative to the chosen fixed time in the past). That issue is being resolved in bug 1713628.

Comment 15

[Rob Wu [:robwu]]

Gregg, can we ensure that Autograph always uses a validity period for root/intermediate certs (used to sign add-ons, i.e. XPI files) that includes a (somewhat arbitrary) choice of verification time (e.g. 1 jan 2021)?
That means that in the future (2025 and beyond) we would sign add-ons with expired intermediates, and/or new intermediates with a very long duration (which chains up to the current root which is expired at that point in the future to have a trusted chain).

For the record:

• The current root is at addons-public.crt and valid from 2015-03-17 until 2025-03-14 (from bug 1038068).
• The current intermediate is at addons-public-intermediate.crt and valid from 2015-04-04 until 2025-04-04 (from bug 1549249).
• End-entity certificates (the temporary certificate used to create the signature for add-ons) used to be valid for one year, now for ten years starting from the time of signing. Expires EE are still accepted (thanks to bug 1267318 aka the first armagadd-on), and EE with a "future" notBefore are accepted once bug 1713628 is fixed.

Comment 16

[Greg G]

(In reply to Rob Wu [:robwu] from comment #15)

Gregg, can we ensure that Autograph always uses a validity period for root/intermediate certs (used to sign add-ons, i.e. XPI files) that includes a (somewhat arbitrary) choice of verification time (e.g. 1 jan 2021)?

We can set NotBefore and NotAfter in generated XPI EEs to include an arbitrary time (patching here). As long as the arbitrary time is within the validity period for the current intermediate and root, we shouldn't need to rotate those certs.

Will we need to resign addons signed since the arbitrary time?

Also, I didn't see it linked here, but the work Julien referenced in comment 7 is in https://github.com/mozilla/renard . And I believe the outcome of the All Hands meeting was that we are not going to pursue the renard v3 signing scheme (not sure if there are notes from that meeting), and it sounds like we're taking an approach similar to V2 signing without expiration.

Comment 17

[Rob Wu [:robwu]]

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #16)

(In reply to Rob Wu [:robwu] from comment #15)

Gregg, can we ensure that Autograph always uses a validity period for root/intermediate certs (used to sign add-ons, i.e. XPI files) that includes a (somewhat arbitrary) choice of verification time (e.g. 1 jan 2021)?

We can set NotBefore and NotAfter in generated XPI EEs to include an arbitrary time (patching here). As long as the arbitrary time is within the validity period for the current intermediate and root, we shouldn't need to rotate those certs.

The EE cert's expiry is already ignored. I'm more concerned about the root and the intermediate cert.

Under the current circumstances (where the "current time" is used to check the validity), the root and intermediate would expire in 2025 and would need to be rotated before that (with an increased notAfter). Firefox builds before this rotation cannot use add-ons any more.

My question was whether it is possible to use the root/intermediate "forever" past the expiry date, which is the case if any of the following is true:

• if Autograph happily signs XPIs with expired non-EE certs (root/intermediate) - ideal case.
• if Autograph doesn't allow an expired cert chain, then it would also be acceptable if we can generate a cert chain with each NotBefore+NotAfter containing the choice of arbitrary time (ultimately ending at a trust anchor that has already been expired but would still accepted by Firefox, i.e. the current root/intermediate).

Can you confirm whether the server side (Autograph) is able to sign XPIs past 2025 with the current (or equivalent) root/intermediate certs?
The client side (Firefox/PSM)

Will we need to resign addons signed since the arbitrary time?

My patch for bug 1713628 ensures that EE with a "notBefore" after the arbitrary time are still accepted.
The proposed patch fixes the verification time to a time where the current root/intermediate certs are still valid.

Also, I didn't see it linked here, but the work Julien referenced in comment 7 is in https://github.com/mozilla/renard . And I believe the outcome of the All Hands meeting was that we are not going to pursue the renard v3 signing scheme (not sure if there are notes from that meeting), and it sounds like we're taking an approach similar to V2 signing without expiration.

Correct. Independently of this patch, v3 signing can still be pursued if there is a good reason for it. The current patch's purpose is to ensure that add-ons continue to run on (older) Firefox versions in the future (past 2025).

Comment 18

[Greg G]

(In reply to Rob Wu [:robwu] from comment #17)

My question was whether it is possible to use the root/intermediate "forever" past the expiry date, which is the case if any of the following is true:

• if Autograph happily signs XPIs with expired non-EE certs (root/intermediate) - ideal case.
• if Autograph doesn't allow an expired cert chain, then it would also be acceptable if we can generate a cert chain with each NotBefore+NotAfter containing the choice of arbitrary time (ultimately ending at a trust anchor that has already been expired but would still accepted by Firefox, i.e. the current root/intermediate).

Autograph doesn't verify the non-EE chain when it handles signing requests (the monitoring and verification code is not happy). However, autograph does check the intermediate cert validity when it starts (which caused problems deploying autograph during armagaddon 2.0). I have a patch in flight to drop that check that I'm planning to release with autograph train-24 (tracking in bug 1720196).

Can you confirm whether the server side (Autograph) is able to sign XPIs past 2025 with the current (or equivalent) root/intermediate certs?

Not yet, but autograph didn't have any issues signing an XPI using a chain of an expired intermediate and expired root in the unit tests for my patch. Once I land that patch I'll bump my system clock and E2E test against the production chain too (golang compiles a static binary so the libfaketime LD_PRELOAD hack doesn't work).

As far as I can tell autograph shouldn't have problems signing XPIs with an expired chain, and we won't need to generate a new one.

Comment 19

[Rob Wu [:robwu]]

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #18)

I have a patch in flight to drop that check that I'm planning to release with autograph train-24 (tracking in bug 1720196).

Looks like https://github.com/mozilla-services/autograph/pull/722 is the WIP patch that you're referring to.

Thanks for clarifying the part about Autograph. I'll discuss this bug with my team tomorrow.

Comment 20

[Tomislav Jovanovic :zombie]

(In reply to Rob Wu [:robwu] from comment #13)

If Autograph is fine with signing XPIs using an expired root certificate, then the patch at https://phabricator.services.mozilla.com/D119812 would be an effective way to avoid armagadd-on-3.0.

Overall this look reasonable, but I don't like it as long term solution. If possible, we should have a plan that allows us to later transition to a (newly-issued) completely valid root and intermediate certificates, that are not back-dated to have a notBefore in the past.

Perhaps that means the hard-coded date is 2025-01-01, and we just need to make sure we mint all new 10 year cert chain before that, or perhaps we could have a second alternative hard-coded date like 2031-01-01 if validation with the first one fails (which might very well be a bad idea for some other reason :).

Comment 21

[Greg G]

(In reply to Rob Wu [:robwu] from comment #19)

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #18)

I have a patch in flight to drop that check that I'm planning to release with autograph train-24 (tracking in bug 1720196).

Looks like https://github.com/mozilla-services/autograph/pull/722 is the WIP patch that you're referring to.

Landed that PR. It pins the EE cert validity to NotBefore 2020-01-01T00:00:00Z and NotAfter 2029-12-29T00:00:00Z (87600 hours) to simplify verification in other code, but it shouldn't impact verification with NSS.

I'll bump my system clock and E2E test against the production chain too

I bumped my system clock ten years in the future to 2031-08-09. Using the prod AMO signer config with the "expired" chain autograph started and signed an XPI with options matching the current prod config (PKCS7 SHA2 digest and an ES256 cose signature).

The PKCS7 SigningTime authenticated attribute is still set to the current clock time here. I don't think it will cause any issues in Fx, but I'll E2E test it against a build with https://bugzilla.mozilla.org/show_bug.cgi?id=1713628#c8 to be sure.

Comment 22

[Rob Wu [:robwu]]

Thanks for your update Greg.

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #21)

(In reply to Rob Wu [:robwu] from comment #19)

(In reply to Greg Guthe [:g-k] [:gguthe] from comment #18)

I have a patch in flight to drop that check that I'm planning to release with autograph train-24 (tracking in bug 1720196).

Looks like https://github.com/mozilla-services/autograph/pull/722 is the WIP patch that you're referring to.

Landed that PR. It pins the EE cert validity to NotBefore 2020-01-01T00:00:00Z and NotAfter 2029-12-29T00:00:00Z (87600 hours) to simplify verification in other code, but it shouldn't impact verification with NSS.

I'll bump my system clock and E2E test against the production chain too

I bumped my system clock ten years in the future to 2031-08-09. Using the prod AMO signer config with the "expired" chain autograph started and signed an XPI with options matching the current prod config (PKCS7 SHA2 digest and an ES256 cose signature).

I intend to also add a unit test for many years in the future (say year 9999), to make sure that verification is not blocked by e.g. the certificate validity being too long. If you think that ten years is a sufficient test, carry on.

The PKCS7 SigningTime authenticated attribute is still set to the current clock time here. I don't think it will cause any issues in Fx, but I'll E2E test it against a build with https://bugzilla.mozilla.org/show_bug.cgi?id=1713628#c8 to be sure.

The signingTime attribute is used by the add-ons blocklist implementation. If you haven't done so yet, please update the comment and/or unit test to make sure that signingTime reflects the time of signing.

Comment 23

[Rob Wu [:robwu]]

I'm exploring the feasibility of using a fixed time to verify add-on signatures to avoid verification failures in the future.
The purpose of this work is to prevent add-ons from being disabled in 2025 due to inaction on our part, and the patch is at https://phabricator.services.mozilla.com/D119812 (tests will be added in the future after the chosen method is confirmed).
Thanks to Gregg, Autograph is prepared to generate signatures with (in the future) expired intermediate / root certificates.

This is effectively the approach described by V2 signing without expiration (using a fixed timestamp is comparable to disabling expiration checking).

Dan - could you take a look and share your thoughts? Is this approach something that we should pursue, or can you think of serious issues against it, with better alternatives?

Comment 24

[Rob Wu [:robwu]]

We recently had a brief incident where some add-ons became disabled because Autograph had set notBefore to a date in the past, while add-on code relied in it: https://github.com/mozilla-services/autograph/issues/739

In comment 22, I wrote the following:

The signingTime attribute is used by the add-ons blocklist implementation. If you haven't done so yet, please update the comment and/or unit test to make sure that signingTime reflects the time of signing.

This is incorrect. While signingTime is in the certificate, it is not exposed via the nsIX509Cert, hence I did not use the signingTime attribute but the notBefore in the implementation, which coincidentally had the same/expected value (when I introduced the date check in bug 1620621).

If the signingTime attribute in the past does indeed mirror the actual signing time (which seems likely and also be the case when I just checked a recent add-on and one from 2017), then it would be possible to update the add-on manager's code to read the signingTime attribute from the certificate (if nsNSSCertificate.cpp exposes a way to do so). There would not be a difference today, but it would allow Autograph to have a different notBefore time in the future if desired. For the addons/blocklist code, the use of notBefore or signingTime is equivalent (except there is no method that reads signingTime). Semantically, using signingTime sounds better than notBefore.

Greg, do you know if there is any value of switching to signingTime instead of notBefore?

Comment 25

[Greg G]

Greg, do you know if there is any value of switching to signingTime instead of notBefore?

It makes testing easier, autograph could use a different notBefore and verify the cert chain with golang's x509 library. I mentioned it in passing in https://bugzilla.mozilla.org/show_bug.cgi?id=1549018#c21 as my reason for pinning the EE cert validity.

That said, one of my takeaways from the incident is that testing addons installs E2E is what matters.

Comment 26

[Rob Wu [:robwu]]

Apparently the above needinfo is "Open bugs with no activity for 2 weeks and a r+ patch which hasn't landed" per https://github.com/mozilla/relman-auto-nag/issues/1739

The patch with a proposal has been approved by Dana, but needs unit tests before landing.
Before landing at all, there needs to be internal agreement that this is the path to take to resolve the bug.

Comment 27

[BugBot [:suhaib / :marco/ :calixte]]

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit BugBot documentation.