1680372 - Add new Let's Encrypt intermediate certificates to mozilla_services pinset

Status: RESOLVED FIXED

(Core :: Security: PSM, task, P1)

Description

[Jon Buckley [:jbuck]]

Let's Encrypt has started using their new Let’s Encrypt R3 intermediate certificate to issue certificates. https://delivery-checks.services.mozilla.com/ has a freshly issued certificate from the R3 intermediate that fails to load in Firefox because the R3 intermediate certificate isn't in the mozilla_services pinset. I'd like to add this R3 intermediate certificate to the pinset so we can continue to use Let's Encrypt.

:dkeeler - Are there any additional steps or approvals needed to add these intermediate certificates to that file? Looks like I need to add the name of each intermediate to the mozilla_services pinset, and the PEM of each intermediate to the extra_certificates key

Also, which intermediates should we be adding? From https://letsencrypt.org/certificates/ it looks like we should be adding R3, R4, E1, and E2.

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler]]

I take it the comment at https://searchfox.org/mozilla-central/rev/de782976bf97669f1e8edee59e7a2398154efe06/security/manager/tools/PreloadedHPKPins.json#77-78 is now out of date? Now that I think about it, can we pin to the Let's Encrypt roots directly? That way we wouldn't need to mess with intermediates.
But in general, the process is whoever's responsible for Mozilla's certificates tells me what issuers we use and then we make sure the mozilla_services pinset accurately reflects that.

Comment 2

[Jon Buckley [:jbuck]]

(In reply to Dana Keeler (she/her) (use needinfo) (:keeler for reviews) from comment #1)

I take it the comment at https://searchfox.org/mozilla-central/rev/de782976bf97669f1e8edee59e7a2398154efe06/security/manager/tools/PreloadedHPKPins.json#77-78 is now out of date?

Yeah, we started using Let's Encrypt certificates for some production services after adding the CAA record and removing the blacklist on the LE side.

Now that I think about it, can we pin to the Let's Encrypt roots directly? That way we wouldn't need to mess with intermediates.

My only Q here would be why did we pin to the specific intermediates for Let's Encrypt in the first place? We're pinning to Digicert roots within the pinset.

But in general, the process is whoever's responsible for Mozilla's certificates tells me what issuers we use and then we make sure the mozilla_services pinset accurately reflects that.

I'll get some secops folks to weigh in here

Comment 3

[J.C. Jones [:jcj] (he/they)]

We pinned to the intermediates because at the time LE was still generally pathbuilding to their IdentTrust root. At this point I think Firefox can pin the ISRG X1 and X2 roots: https://letsencrypt.org/certificates/

Comment 4

[Sven Marnach :sven]

Since the retired intermediates are also signed by the X1 root, we should be able to remove all intermediates from the set as well, so this sounds like the easiest and most robust option.

Comment 5

[Sven Marnach :sven]

It looks like the "ISRG Root X2" cert isn't included in Firefox's root store yet, so would we already be able to include it in the pin set?

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Not without including its contents like with intermediates. It's probably not a good idea to pin to a root that isn't in our root program.

Comment 7

[Sven Marnach :sven]

It looks like the X2 cert isn't currently used, and all intermediates in current use are signed by the X1 cert, so it would be enough to include only that cert.

Comment 8

[Aaron Gable]

Correct, we (Let's Encrypt) aren't issuing from E1 or E2 yet, which are the only intermediates issued from ISRG Root X2 thus far. We'll be submitting X2 to root programs in early 2021. And even when we do begin issuance, ISRG Root X2 has been cross-signed by ISRG Root X1, so pinning X1 should still be sufficient for a while.

Comment 9

[Sven Marnach :sven]

It would be nice to get this change into the next release. Soft freeze is on Thursday, so this looks rather tight.

Not getting this fix in is risky, since we don't know which services in the services.mozilla.com domain already use Let's Encrypt. Service that do use Let's Encrypt may issue new certs that will no longer be accepted by Firefox, which can break important functionality. I will try to find out whether there are any such services. At the same time, we should try and land a patch in mozilla-central as soon as possible. We can then still decide whether we want to request an uplift.

Do we need any additional information, or are we confident that removing the intermediates and adding the X1 root certificate is the right course of action?

Comment 10

[Sven Marnach :sven]

Let's Encrypt is also in the pin set for

• some torproject.org domains (but that pin set includes the R3 and R4 intermediates)
• swehack.org (which does not include the R3 and R4 intermediates, but currently doesn't seem to use Let's Encrypt; also WTF?)

So we should update these as well (or consider removing the latter).

Comment 11

[Sven Marnach :sven]

We had a few discussions within SRE about this. Here's a summary of some of the findings.

Expected imapct

We have pins for Let's Encrypt certificates for the following domains:

• *.sevices.mozilla.com
• some torproject.org domains
• swehack.org

Only the first bullet point should pose a problem. The pin set for the torproject.org domains includes the R3 and R4 intermediate certs, so we don't expect any problem there. The swehack.org domain pin set does not inckude the newer intermediate certs, but it doesn't currently use Let's Encrypt. (It's also rather bizaree that we have a pin for that domain in the first place.)

As for the *.services.mozilla.com domain, we haven't completed the survery of affected domains in our team. We've identified two so far that could cause problems:

• stubdownloader.services.mozilla.com
• cdn.stubdownloader.services.mozilla.com

These are related to automatic Firefox updates.

Possible solutions

Given that there seems to be a rather small number of affected domains, and that it's basically too late to get anything into the release, the most reasonable course of action seems to be to migrate the affected domains back to Digicert for now.

We should also get a fix into Nightly as soon as possible, so we can start testing the fix.

Open questions

• I remember Julien Vehent told me that the pins expire some weeks after release. How exactly does this expiry work? When will older versions of Firefox stop respecting these pins? What about ESR releases? :keeler, would you be able to comment on this?

• What is the full list of domains under services.mozilla.com that is potentially affected? I'll file a separate bug to work this out.

Comment 12

[Dana Keeler (she/her) (use needinfo) [:keeler]]

For context, we automatically incorporate some pinsets from Chromium, but not all of them are enabled. For example, the swehack.org pinset isn't enabled [0] (the second boolean being true means it's in test-only mode). Also, modifying the mozilla_services pinset will have no effect on other pinsets or on domains not using that pinset.

The expiration time for pinsets can be found at [1]. The timestamp is in microseconds since the epoch, so new Date(timestamp / 1000) will give you the timestamp in JS. For example, it looks like the pins in Firefox 83 will expire on February 11th. The expiration times are updated when the automated update scripts run ([2]). This means that as long as we keep updating ESR, it will keep having its pinning expiration date extended. When the pins expire, they are ignored, so it's as if pinning has been disabled completely.

Any entry in the list that starts at [3] with the attribute pins: mozilla_services will be effected by changes to that pinset. The first boolean in each entry indicates whether or not subdomains will also be affected.

[0] https://searchfox.org/mozilla-central/rev/1843375acbbca68127713e402be222350ac99301/security/manager/ssl/StaticHPKPins.h#1078
[1] https://searchfox.org/mozilla-central/rev/1843375acbbca68127713e402be222350ac99301/security/manager/ssl/StaticHPKPins.h#1145
[2] https://hg.mozilla.org/mozilla-central/log/7167ab3f6e8d7b77439ac1cc7d9356bbaff02510/security/manager/ssl/StaticHPKPins.h
[3] https://searchfox.org/mozilla-central/rev/1843375acbbca68127713e402be222350ac99301/security/manager/tools/PreloadedHPKPins.json#143

Comment 13

[Sven Marnach :sven]

Thanks for the explanations! I had missed that the mozilla.services.com pinset is in use for many different domains. So here is the full list:

650:  { "accounts.firefox.com", true, false, true, 4, &kPinset_mozilla_services },
652:  { "addons.mozilla.net", true, false, true, 2, &kPinset_mozilla_services },
653:  { "addons.mozilla.org", true, false, true, 1, &kPinset_mozilla_services },
656:  { "api.accounts.firefox.com", true, false, true, 5, &kPinset_mozilla_services },
664:  { "aus4.mozilla.org", true, true, true, 3, &kPinset_mozilla_services },
665:  { "aus5.mozilla.org", true, true, true, 7, &kPinset_mozilla_services },
681:  { "cdn.mozilla.net", true, false, true, 16, &kPinset_mozilla_services },
682:  { "cdn.mozilla.org", true, false, true, 17, &kPinset_mozilla_services },
707:  { "crash-reports-xpsp2.mozilla.com", false, false, true, 11, &kPinset_mozilla_services },
708:  { "crash-reports.mozilla.com", false, false, true, 10, &kPinset_mozilla_services },
709:  { "crash-stats.mozilla.org", false, false, true, 12, &kPinset_mozilla_services },
726:  { "download.mozilla.org", false, false, true, 14, &kPinset_mozilla_services },
742:  { "firefox.com", true, true, true, 15, &kPinset_mozilla_services },
1070:  { "services.mozilla.com", true, false, true, 6, &kPinset_mozilla_services },
1079:  { "sync.services.mozilla.com", true, false, true, 13, &kPinset_mozilla_services },
1084:  { "telemetry.mozilla.org", true, true, true, 8, &kPinset_mozilla_services },
1086:  { "testpilot.firefox.com", false, false, true, 9, &kPinset_mozilla_services },

Excluding domains in test mode, this leaves these domains (and possibly their subdomains, as indicated by the first bool in the list above):

*.accounts.firefox.com
*.addons.mozilla.net
*.addons.mozilla.org
*.api.accounts.firefox.com
*.cdn.mozilla.net
*.cdn.mozilla.org
crash-reports-xpsp2.mozilla.com
crash-reports.mozilla.com
crash-stats.mozilla.org
download.mozilla.org
*.services.mozilla.com
*.sync.services.mozilla.com
testpilot.firefox.com

Comment 14

[Sven Marnach :sven]

The pins for the upcoming release will expire on 2021-03-15, both for the ESR update and Firefox 84.

Comment 15

[Sven Marnach :sven]

We have identified only three domains that need to be migrated back to DigiCert, so on our end we are good for now.

We should try and get the pin set change into the January release, though, which means we should still try and change this quickly in Nightly. :keeler, could you please change the Mozilla services pin set to include the X1 root certificate instead of the LE intermediates that are currently there? If we get this landed in the next few days, we may not even have to ask for a beta uplift, but I don't really no what the timelines for that are.

Once the new X2 root cert is in the Firefox root set, we should add that cert as well, but that's out of scope for now.

Comment 16

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: Bug 1680372 - replace Let's Encrypt intermediate certificates with ISRG Root X1 in the mozilla_services pinset r?kjacobs

Now that we're actually using Let's Encrypt for Mozilla services, we should pin
to the root.

Comment 17

[Sven Marnach :sven]

Thanks! After this change lands, when will StaticHPKPins.h be regenerated? Or in other words, when will the change be available for testing in Nightly?

Comment 18

[Dana Keeler (she/her) (use needinfo) [:keeler]]

After landing, the changes should be available the next time the automatic updates task gets run, which I believe is daily.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e73f47196a5d
replace Let's Encrypt intermediate certificates with ISRG Root X1 in the mozilla_services pinset r=kjacobs DONTBUILD

Comment 20

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/e73f47196a5d

Comment 21

[Sven Marnach :sven]

Dana, I still get a SEC_ERROR_UNKNOWN_ISSUER when trying to navigate to https://delivery-checks.services.mozilla.com/ in Nightly. Is this expected? I would have expected that the change should have made it into Nightly by now.

Comment 22

[Sven Marnach :sven]

I just got a Nightly update, and now it works! We will continue testing this to make sure this works fine.

Comment 23

[Sven Marnach :sven]

This change is now also available in the latest beta release, so we don't need to request an uplift: https://hg.mozilla.org/releases/mozilla-beta/file/6bfff9aa8e67619a8173ac4eab1cd6b54b495419/security/manager/ssl/StaticHPKPins.h#l435

Comment 24

[Sven Marnach :sven]

The change did not make it into the ESR branch, though. I'll try to figure out how we can get it in, or whether we even need to do anything.

Comment 25

[Sven Marnach :sven]

Comment on attachment 9192306 [details]
Bug 1680372 - replace Let's Encrypt intermediate certificates with ISRG Root X1 in the mozilla_services pinset r?kjacobs

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: The mozilla_services pin set is outdated; Let's Encrypt now issues certificates from different intermediate certificates. This means we currently can't use Let' Encrypt certs for any domain name using the mozilla_services pin set, which increases operational effort.

It's possible the pin sets are uplifted as part of the regular release process, in which case this request is moot.

• User impact if declined: No impact for users, but increased effort for the ops teams.
• Fix Landed on Version: Nightly, beta
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Messing up the mozilla_services pin set can in theory result in clients not being able to reach any Mozilla services anymore (including further updates), so in general this may be considered risky. However, we tested this successfully in Nightly, and we can confirm that the new pin set works as expected.
• String or UUID changes made by this patch:

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9192306 [details]
Bug 1680372 - replace Let's Encrypt intermediate certificates with ISRG Root X1 in the mozilla_services pinset r?kjacobs

Approved for 78.7esr.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/eef85e163995