1742195 - Microsoft PKI Services: Failure to disclose Revocation of Intermediate CAs within 7 Days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[John Mason]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36 Edg/95.0.1020.53

Comment 1

[John Mason]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Microsoft PKI Services was recently assigned this Bugzilla Bug (http://bugzilla.mozilla.org/show_bug.cgi?id=1740585) Microsoft: Unrevoked 4 intermediate certificates. During investigation of this bug, we realized that the CCADB listing of these four (4) ICA certificates did not indicate they had been revoked. We felt they should be updated and did just that on 13 Nov 2021. The ICA’s were revoked on 24 June 2021. They were manually loaded into CCADB on 31 August 2021 by a CCADB Administrator.

We feel that our change to the status in CCADB so far after the fact may be a violation of the Mozilla Root Store Policy to upload changes to certificates in CCADB within 7 days. Although there are extenuating circumstances to consider, and we would welcome comments on this issue.

2. A timeline of the actions your CA took in response.

Note: Times are listed in the Pacific time zone.

24 Mar 2021 - Microsoft PKI Services opened a Bugzilla bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1700809) for Microsoft PKI Services: Failure to disclose Unconstrained Intermediate within 7 Days. This bug resulted in updated processes by our team posting to CCADB in a timely manner.

02 Jul 2021 12:50 PM - Microsoft PKI Services opened a Bugzilla bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1718991) for creating 4 Malformed ICA's. This bug explains how the ICA's were created and revoked. The 4 ICA's from this new Bug are the same 4 ICA's that were Malformed in this original bug. Our procedures had us trying to post these to CCADB, but we ran into problems uploading the Certificates to CCADB (due to their malformation) and contacted CCADB Administrators for help.

07 Jul 2021 5:18 PM - Kathleen Wilson comments in the bug (1718991) that explained trepidation about adding the PEM files for these malformed certs to CCADB (https://bugzilla.mozilla.org/show_bug.cgi?id=1718991#c7). The Certs were not added to CCADB at the time, and we felt that we had properly notified and disclosed Mozilla and the community about the Malformed CA’s and the issues related to disclosing in CCADB.

10 Nov 2021 11:23 PM – Kathleen Wilson from Mozilla opened a Bugzilla Bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1740585) for Microsoft: Unrevoked 4 intermediate certificates and it was assigned to Microsoft PKI Services. This bug had us revisit the status of these four (4) ICA’s in CCADB.

13 Nov 2021 6:35 PM - Corrected the CCADB entries for the CAs to Revoked status

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

Microsoft PKI did not issue any certificates from the CAs as the CA certificates were revoked immediately after detecting the issue with the malformed certificates.

4. In a case involving certificates, a summary of the problematic certificates.

There are four (4) ICA’s involved with this issue. They were all created and revoked on 24 June 2021.
https://crt.sh/?sha256=ec02314a59a303990772bdf25513d5093581257ad4e242f086f988a98fba8b7d
https://crt.sh/?sha256=3acd6f50d569963ede389e5a3d024fef52cb537dbf497ca1725e9ce710117807
https://crt.sh/?sha256=eb79c04645b9137e67647a7389dac6eb1d3aad8aa74d8994aa8f9c01015ecde0
https://crt.sh/?sha256=3b7d95d4ff780b5ea537d852e24c5485cda83b2b7931c7af1c8feec9c62146db

5. In a case involving certificates, the complete certificate data for the problematic certificates.

See list of certificates above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As described in the timeline of events above, this started when four (4) Malformed ICA’s were created on 24 Jun 2021 and this was disclosed in Bugzilla Bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1718991). The four (4) ICA certificates were so malformed that they could not be uploaded to CCCADB.

After conferring with the CCADB Administrator about the issue, Kathleen Wilson, it was decided that the certificates would not be uploaded to CCADB. That was also disclosed and discussed in the Bug (1718991) by Kathleen on 7 Jul 2021.

At this point, we felt that we had met the Mozilla Root Store Program requirement to post in 7 days (as we had come to a resolution that seemed to work).

On 31 Aug 2021 Microsoft PKI Services became aware that these same ICA’s were showing up in the Mozilla CA Certificate Disclosures Report. We contacted the CCADB Administrator again to discuss how to handle this. It is at this point that the CCADB Administrator manually uploaded the four (4) ICA certificates to CCADB (31 Aug 2021).

This meant that the report was no longer reporting these ICAs as noncompliant.

At this point, we had not noticed that the certificates were not marked as revoked in CCADB (31 Aug 2021).

While investigating the newer incident (1718991) we noticed that the four (4) ICAs were not marked as revoked in CCADB and on 13 Nov 2021 we rectified the status of all four.

7. List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

As mentioned above, we updated the CCADB records on 13 Nov 2021 to indicate the revocations. We feel that we did meet the Mozilla Trust Root Programs requirements to update CCADB, as we did disclose and come to a mutual agreement within 7 days of the creation of these four (4) certificates and the revocation of them (all occurred on 24 June 2021).

However, given the circumstances where they were finally manually uploaded by the CCADB Administrator on 31 Aug 2021 and the correct status was not reflected until 13 Nov 2021 we wanted to disclose this issue and open it to discussion from the community.

All of this was no doubt aggravated by the circumstances described in Bugzilla bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1718991) for creating 4 Malformed ICA's. In that Microsoft PKI Services inadvertently unrevoked these four ICA’s.

As discussed in (https://bugzilla.mozilla.org/show_bug.cgi?id=1700809) for Microsoft PKI Services: Failure to disclose Unconstrained Intermediate within 7 Days, Microsoft PKI Services is evaluating automating CCADB updates using the tools available to the community. While it will still be sometime until we have that capability, we foresee future functionality to automatically check that our records and the CCADB records are in synchronization and that could help prevent errors like this in the future.

Comment 2

[John Mason]

Please let us know if anyone has any comments on the above incident, as we would appreciate the feedback.

If there are no comments, then we would ask that this incident please be marked as resolved.

Comment 3

[Ben Wilson]

I'll close this on Friday, 3-Dec-2021, unless there is further discussion that needs to take place.