1586847 - Microsoft: Issuance of intermediates after 2019-01-01 that do not comply with Mozilla Policy

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

During a spot-check of Mozilla Policy Compliance, I discovered that Microsoft issued an intermediate that does not conform with Mozilla Policy 2.6.1

Mozilla Policy 2.6.1 requires (formatted for readability), in Section 5.3:

Intermediate certificates created after January 1, 2019, with the exception of cross-certificates that share a private key with a corresponding root certificate:

• MUST contain an EKU extension; and,
• MUST NOT include the anyExtendedKeyUsage KeyPurposeId; and,
• MUST NOT include both the id-kp-serverAuth and id-kp-emailProtection KeyPurposeIds in the same certificate.

The intermediate is the "Microsoft TLS EV Issuing CA 02", https://crt.sh/?q=623a0d3c6e325795aadaa36599301afd90392f46519f5b238fbe22936bbd5ccc , issued 2019-05-14 under the "Microsoft EV RSA Root Certificate Authority 2017"

This certificate lacks the extendedKeyUsage extension.

Per Comment 68 on Bug 1448093, please provide an incident report, as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident

Comment 1

[Jason Cooper]

We renewed this CA to include the required EKUs but it doesn't look like that renewed cert was published to AIA. We're working on this now and will report back when complete.

Comment 2

[Jason Cooper]

Attachment: Incident Report - Bug 1586847

Attached is the Incident Report for Bug 1586847. Please let me know should additional information be needed or if you have questions.

Comment 3

[Jason Cooper]

I have attached a Word document with our incident report. Please let me know if you prefer a different format for the report.

Comment 4

[Ryan Sleevi]

Hi Jason: Could you provide the answers inline here? Bugzilla supports Markdown formatting, if you'd like to format it, but this also helps ensure the answers are clear, fully searchable, and easily accessible.

Comment 5

[Jason Cooper]

Here is the text from attachment.

Mozilla Team, please let me know if you have any questions or would like additional details.

CA Awareness and Timelines:

On May 14th and June 11th, 2019, the Microsoft PKI Services team created the ten CAs listed in the below table. The CAs were errantly created without the Client and Server Authentication EKUs as required by Mozilla policy. This was noticed by members of the Microsoft PKI Services team and plans were made to renew these errantly issued CAs. Renewed versions of all ten CAs were created on July 25th, 2019.

On October 7th, 2019, Mozilla incident bug 1586847 was created, noting that “Microsoft TLS Issuing CA 02” was missing the required EKUs.

On October 14th, 2019, Microsoft revoked the initial versions of all ten CAs. The updated CRLs can be viewed at the Microsoft PKI Services Repository: https://www.microsoft.com/pkiops/docs/repository.htm

List of out of policy CAs (now revoked) with Common Name, Serial Number, SHA-1 Thumbprint, and Valid From Date:

Microsoft TLS Issuing CA 02
330000000C1DFF833EC83C2B7400000000000C
935E393FB5BC5251D20D6A6D6E470B66205C7C7B
5/14/2019

Microsoft TLS Issuing CA 03
330000000D5AFDB3E4503C70E200000000000D
097BC45A17C9478BCE4118CED1872F5AB37FFAF5
5/14/2019

Microsoft TLS Issuing CA 06
330000000E1EDF9B609EFAFF0100000000000E
EF381850F3144468AC01876AA69C48ABEEB86FC8
5/14/2019

Microsoft TLS Issuing CA 07
330000000FD5F279C64DE3514600000000000F
923DFE400E3341CE74C90F1EBABAE70D92E9F4D4
5/14/2019

Microsoft TLS ECC Issuing CA 02
330000000A821726907135A3F600000000000A
742058DB6851B0751A5256F9AE3727A346963773
5/14/2019

Microsoft TLS ECC Issuing CA 06
330000000BC3416D6211DC9D2A00000000000B
C197C537828FF9105093B6869DBA418ED9C90A33
5/14/2019

Microsoft TLS EV Issuing CA 02
330000000EE7E0958C8A43FC9000000000000E
ACB5105A1C8CAD653E539E8E1C7020BED02BC03F
5/14/2019

Microsoft TLS EV Issuing CA 06
330000000FEBE2B7FD288667A300000000000F
56F7DB8B1C23E3CE1F6E49246DE66D970962DCF1
5/14/2019

Microsoft TLS ECC EV Issuing CA 02
33000000081B8CDADDB6EE2B45000000000008
3E1FF20C6DEB5CB8987AB45FB9D343C2EF00270D
6/11/2019

Microsoft TLS ECC EV Issuing CA 06
33000000091F7F3773EA9D4699000000000009
CAE747454EBE545EAE078363E9DFE5D60371965B
6/11/2019

List of renewed CAs that include the required EKUs:

Microsoft TLS Issuing CA 02
33000000140E723766CFB1420F000000000014
DD9D0106A59F2A4CB47C99AE43379B6E2E1AA250
7/25/2019

Microsoft TLS Issuing CA 03
3300000015085DCE13507604C8000000000015
97ACA404562EAAAD0EF00F71F041F3E772A568E4
7/25/2019

Microsoft TLS Issuing CA 06
3300000016C097F3161C6C9AE5000000000016
4A51A19759A5CBFDD5424CBCC803C266C2F04441
7/25/2019

Microsoft TLS Issuing CA 07
33000000178F9C5D4DEC164F63000000000017
422147C198301E00B5F97A793EDF8EB2E291971C
7/25/2019

Microsoft TLS ECC Issuing CA 02
330000001425815B5CB1CCE0C1000000000014
E775371E06C1A13CB0F4B22F05BE3070A86B0074
7/25/2019

Microsoft TLS ECC Issuing CA 06
3300000015889C316A4742299A000000000015
905DD89F0166AE61240AA56D8D2A9803A695AF94
7/25/2019

Microsoft TLS EV Issuing CA 02
33000000101765FC11EB5F4B26000000000010
06ABABABDF79C7FCEBFC841A0218A2D14AE8BFF3
7/25/2019

Microsoft TLS EV Issuing CA 06
330000001127CBE118C4847C71000000000011
948711C4552D93A912263578E67B69FE3E47EF57
7/25/2019

Microsoft TLS ECC EV Issuing CA 02
330000000A0FBCE38F00264B3300000000000A
369E91A5A841231BE477AE44663553A52FB69A2D
7/25/2019

Microsoft TLS ECC EV Issuing CA 06
330000000B854D7E490E80DC9E00000000000B
214733F95D6C0BF0E6FAC642F1BC25C0242AEEFF
7/25/2019

Summary of Problematic Certificates:

There are no valid problematic subscriber certs from either the initial versions or renewed versions of these CAs.

Root Cause and Preventative Measures:

CA Ceremony materials for Intermediate CAs in this category (Cross-Platform Public Trust) were not up to date at the time of these errors. They are now updated and any future intermediate CAs will include the required EKUs.

Comment 6

[Ryan Sleevi]

Thanks Jason.

While this is helpful to understand the "what" happened, I'd be remiss if I didn't highlight that the "why" and "how it's being prevented" leave a lot to be desired.

While I want to acknowledge that, as best I can tell, at the time of the incident Microsoft was pending inclusion, one natural question that emerges is "What other procedures or ceremonies may exist that do not comply with Mozilla Policy?" This is why the 'why' is important - understanding the processes that existed before the issue, and how not only been changed, but how they've been reviewed for other systemic issues, is useful.

I also greatly appreciate the disclosure that there were many more certificates. That's definitely a good sign in providing a holistic incident report. One thing from the incident report is wanting to make that "The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.". The information provided doesn't really have that. crt.sh has a submission assistant that can help you construct the JSON to log these intermediate certificates to CT logs - both the revoked and the active certificates.

In the spirit of being consistent with other incident reports, we expect CAs to use the template in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report , as that helps make sure all the information is present, and things like this (the complete certificate details) aren't overlooked.

Finally, it's important and useful to understand why the original problematic certificates, though detected, weren't revoked at the time of detection. What sort of decision making was contemporaneous at that time? Similarly, the decision to introduce new certificates with the same subject and SPKI is certainly interesting. While the policy does not forbid it, it's surprising because certain products, including Microsoft's, have been known to have pathologies when there exists two versions of a certificate for the same named entity. While typically this is done to avoid breaking existing certificates, the freshness of Microsoft's CA makes it seem unlikely that would be as significant a consideration.

As Microsoft gets up to speed on incident responses, it might be worth reading a bit more about the goals and to look through some of the good examples.

To recap some of the specific questions:

• Please provide a report that clearly addresses each of the enumerated items from the incident response.
• Please make sure to provide the full certificate data for all the certificates.
• Please provide more context and explanation for the Cross-Platform Public Trust ceremony issues.
  • When were the current ceremonies developed?
  • How are those ceremonies maintained?
  • How are those ceremonies reviewed prior to issuance?
  • What sort of tools/controls exist to make sure those ceremonies are correct?
  • What's changed, process wide, from prior to this incident, to help make sure future changes are similarly incorporated?
• Please provide more details about why these certificates were not revoked when originally detected?
  • What sort of policies/expectations existed at that time?
  • What sort of changes have been made to those policies since then?

The idea here is to provide sufficient detail that, if you were reading a report from another CA, you'd be able to understand the issue, understand how it might affect your organization, and have a good idea about best practices that might prevent this. I think this is especially important, given that we've seen a few CAs recently have issues with their manual processes. Going back to cover the past 3-5 years, and we see CAs who completely botched manual ceremonies (e.g. during the SHA-1 exception process), and so we know this is a tough area. There have been a lot of ideas suggested, and some of the CAs have implemented them, so what sort of review of those is being done, what's Microsoft's view on best practice to prevent these, and why didn't the existing controls catch this?

Thanks!

Comment 7

[Wayne Thayer]

Jason: it appears that the original versions of these CA certificates have not been disclosed in CCADB? Please disclose and mark them as revoked [1]. And please be aware that the Mozilla policy requiring disclosure of intermediate certificates within one week of creation now applies to Microsoft as a member of our root program.

[1] https://www.ccadb.org/cas/intermediates#marking-an-intermediate-certificate-as-revoked

Comment 8

[Jason Cooper]

Hi Ryan,

I wanted to let you know that I'll be submitting more details shortly and we will also push our intermediates to CT.

Comment 9

[Jason Cooper]

Thanks Wayne. CCADB is now updated to include the revoked CAs.

Comment 10

[Jason Cooper]

Here is an updated report that I hope better captures the incident and comments/questions

Please let me know if there are further questions

1. How your CA first became aware of the problem
   On May 14th and June 11th, 2019, the Microsoft PKI Services team created the ten CAs listed below and attached to this bug as an Excel file. The CAs were errantly created without the Client and Server Authentication EKUs as required by Mozilla policy. This was noticed by members of the Microsoft PKI Services team and plans were made to renew these errantly issued CAs. Renewed versions of all ten CAs were created on July 25th, 2019.

2. A timeline of actions your CA took in response
   The Microsoft PKI Services team noticed the error and renewed the CAs on July 25th.
   On October 7th, 2019, Mozilla incident bug 1586847 was created, noting that “Microsoft TLS Issuing CA 02” was missing the required EKUs.
   On October 14th, 2019, Microsoft revoked the initial versions of all ten CAs. The updated CRLs can be viewed at the Microsoft PKI Services Repository: https://www.microsoft.com/pkiops/docs/repository.htm

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

There are no valid problematic certs related to this problem. Any future certs will come from compliant intermediate CAs.

4. A summary of the problematic certs.

Ten CA certs errantly created without the EKU constraints required by Mozilla as of January 1, 2019.

5. The complete certificate data for the problematic certificates.

List of revoked CAs that did not include the required EKUs:
https://crt.sh/?id=2025778833
https://crt.sh/?id=2025778249
https://crt.sh/?id=2025776988
https://crt.sh/?id=2025777000
https://crt.sh/?id=2025778059
https://crt.sh/?id=2025778819
https://crt.sh/?id=1587694780
https://crt.sh/?id=2025776979
https://crt.sh/?id=2025778299
https://crt.sh/?id=2025776985

List of renewed CAs that include the required EKUs:
https://crt.sh/?id=2004589131
https://crt.sh/?id=2004588867
https://crt.sh/?id=2004588702
https://crt.sh/?id=2004588507
https://crt.sh/?id=2004589105
https://crt.sh/?id=2004589060
https://crt.sh/?id=2004588641
https://crt.sh/?id=2004588508
https://crt.sh/?id=2004588724
https://crt.sh/?id=2004588754

6. Explanation about how the mistakes were made or bugs introduced, and how they avoided detection until now.

The mistakes were made due to processes that had not been updated to account for the policy change introduced by Mozilla effective January 1, 2019. Specifically, the mistake was in a CA ceremony that includes both automated and manual steps that was executed by multiple trusted role personnel. At the time our ceremony template for this category of CA, “Cross-Platform Public Trust”, did not include the steps to add the Client and Server Authentication EKUs. The simplest explanation is, while we have extensive experience executing similar ceremonies, we had not been required to EKU constrain intermediate CAs prior to our application to Mozilla. That said, this was a mistake in our process, and it has been fixed via adjustments to our template.

We have ceremonies for many different CA creation activities. They were developed and are maintained by the Microsoft PKI Services team with appropriate oversight and feedback from other Microsoft employees (i.e. Microsoft PKI Policy Authority [governance], Windows Certificate Services subject matter experts, security assurance professionals, etc.) and 3rd parties such as auditors.
Ceremonies are reviewed by multiple trusted role personnel prior to execution.

They are versioned by Microsoft PKI Services trusted role personnel and maintained in restricted access locations.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Fortunately, the change needed to prevent future incidents was relatively simple as an updated ceremony template now exists. The ceremony template has been utilized since being updated validating the desired result of including the required EKU constraints. Additionally, trusted role personnel responsible for CA operational work, including CA ceremonies, have been trained on this and other Mozilla specific policies.

Additional information:

Regarding timelines and decision making for detection, response, and revocation, these CAs were built for future capabilities, were not part of the Mozilla program at the time, and with no indication of a risk to consumers or a compromise, etc., we made the decision to do the work around existing operational work schedules. Ryan mentioned some potential downsides to renewals, and while we recognize those potential downsides, in this case the renewal made sense to us due to the fact that this was new PKI intended for future use. In retrospect, despite the lack of risk to consumers or other relying parties, revoking the unneeded CA certs quickly would have been a cleaner approach.

Comment 11

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.