1647084 - DigiCert / Microsoft: inconsistent disclosure of externally-operated intermediate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

DigiCert has disclosed the following intermediate as being operated under the same CP/CPS as parent:

https://crt.sh/?sha256=3458C7F787C30FE9AF3E58A7AB6877B1959AB7CC2C4581B070BDD38C39B84AF7

However, the same subject + SPKI is found in this intermediate, which Microsoft has disclosed as being operated under their CP/CPS:

https://crt.sh/?sha256=2CAEFBB55E70DF5A8985FE9BC10DD56A40C3DEDAB3DA1530A29682015C5B7C66

The SPKI is listed in the following Microsoft audit report, so I assume the subject of these intermediates is really operated by Microsoft, and DigiCert has disclosed the DigiCert-issued intermediate incorrectly: https://www.microsoft.com/pkiops/docs/Content/seals/Microsoft%20WTCA%20Indp%20Acct%20Opinion%20and%20Mgmt%20Assertion%20Dec%202019%20-%20Final%20-%20SECURED.pdf

Comment 1

[Ben Wilson]

If this is a check-box, inputting error into the CCADB, and it is corrected promptly, should it still be considered an incident requiring an incident report?

Comment 2

[Ryan Sleevi]

I believe yes, it should be. This has been a persistent pattern with DigiCert, and representing the audits and operational control are an important part of Mozilla’s policy, in Section 8.

It should not require independent members of the public to highlight CAs’ disclosure failures, especially when tools exist to detect this. We should holistically consider how well a CA performs, especially when it comes to public commitments.

Comment 3

[Brenda Bernal]

I am acknowledging receipt of this report, and can confirm the entry error found is correct. We are investigating and will come back with an incident report shortly.

Comment 4

[Brenda Bernal]

Incident Report – Mozilla Policy Violation

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were notified through this opened bug (1647084). Thank you for bringing this to our attention.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

02/20/20 - The request sent from MS for the ICA signing.
04/29/20 – The naming docs were finalized.
05/20/20 - The ICA was created.
05/21/20 - The ICA uploaded to CCADB. The PKI Ops personnel marked same as parent.
06/20/20 - Bug 1657084 was filed notifying DigiCert of the potential error in the CA markings in CCADB.
06/20/20 - Investigation began. Compliance team starts monthly comprehensive sweep of CCADB for review of anomalies.
06/21/20 – We acknowledged the error; CCADB entry was updated for the ICA.

3. Whether your CA has stopped, or has not yet stopped, issuing certificateswith the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We are holding off on any new DigiCert SubCA signings until we complete our investigations and our remediation is in place.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

ICA (issued on 5/20/2020): https://crt.sh/?sha256=3458C7F787C30FE9AF3E58A7AB6877B1959AB7CC2C4581B070BDD38C39B84AF7

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See 4) above

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The issue was a mistake made by our PKI Operator in assuming that this ICA will be hosted by us and marked the ICA “Same as Parent” during the CCADB upload process. This is the first time for our PKI Operator in handling a cross signing ceremony for an external entity. Because we use the same request documents to create both external and internal ICAs , there was no distinguishing way for the PKI Ops team to know this was externally operated. We also rarely create these, allowing TLS SubCAs only for browsers. The general policy combined with the lack of visible indicators that this was different from other signings, meant we followed the standard process for uploading a new ICA to CCADB, rather than the process for external ICA selection.

To resolve this issue, we are going to add the CCADB information to the signing request so it’s clear what the CCADB information should look like before signing the CA. We’re still working out how to better distinguish documentation in creating On Prem and hosted ICAs.

To also note, there was a Compliance review of the cert before release which confirmed the ICA was created as per the approved naming document. Compliance also signed off on the CCADB upload. Similar to PKI Ops, there wasn’t a clear indication that this was an external sub CA and required a different CCADB entry from DigiCert-hosted ICAs.

This audit information would have been corrected in the next annual audit, when the next Microsoft audit lists the ICAs operated by them. That does leave a gap in information between new ICA creation and audit coverage, which the new request documentation will aim to clarify for correct entry into CCADB.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We are re-examining the ICA creation document to make sure we fix the gap with key information missing that would have marked ICA signings as not hosted by us. This will include information about what to upload to CCADB for on-prem CAs and will require signed off by Compliance and the PKI Ops team before ICA creation.

We plan to create a separate naming document that will only be used for SubCAs or Intermediates that will be operated on-prem by the customer. Until this is in place, we are halting any new subCA signings for DigiCert until this is well established.

Comment 5

[Andrew Ayer]

The list of problematic certificates is incomplete. Here's another CA disclosed as being operated by DigiCert:

https://crt.sh/?sha256=6CA3E5D7720215F94544523E3C474B22DA69C732AE455DC82F8F5EE302EC55FC

Yet has also been disclosed as being operated by Microsoft here:

https://crt.sh/?sha256=0437AB2EC2C2B4890296C135034B21DB146434B8317EE703AA8AA943C5EA51AE

Comment 6

[Rob Stradling]

See also https://crt.sh/mozilla-disclosures#disclosedwithinconsistentaudit

Comment 7

[Ryan Sleevi]

Brenda: Could you clarify why Comment #5 was not detected during

06/20/20 - Investigation began. Compliance team starts monthly comprehensive sweep of CCADB for review of anomalies.

From how the timeline is presented, and the subsequent report, it seemed like

06/21/20 – We acknowledged the error; CCADB entry was updated for the ICA.

Was the completion of the comprehensive sweep. Minimally, it would seem like Comment #4 indicated completion of that sweep, per the listing of affected certificates.

Comment 8

[Brenda Bernal]

Hi Ryan,

We had not completed our investigation upon posting of the Incident Report in https://bugzilla.mozilla.org/show_bug.cgi?id=1647084#c4. We rectified the situation right away on the CA identified in 4) above.

As noted in 3), we are holding off on any new DigiCert SubCA signings until we complete our investigations and our remediation is in place.

The team has completed review of CAs uploaded in the last 24 months. As a result, we have found 7 more CAs that had the same errors identified with the CA that was reported for this problem report.

Coincidentally, these were all in the same signing for the same customer – Microsoft Azure.
Here is the list of those ICAs which have been corrected.

Microsoft Azure TLS Issuing CA 01
https://crt.sh/?id=2841732843
Microsoft Azure TLS Issuing CA 02
https://crt.sh/?id=2841732828
Microsoft Azure TLS Issuing CA 05
https://crt.sh/?id=2841732842
Microsoft Azure TLS Issuing CA 06
https://crt.sh/?id=2841732827
Microsoft Azure ECC TLS Issuing CA 01 (subject of this problem report)
https://crt.sh/?id=2841732943
Microsoft Azure ECC TLS Issuing CA 02
https://crt.sh/?id=2841732835
Microsoft Azure ECC TLS Issuing CA 05
https://crt.sh/?id=2841732847
Microsoft Azure ECC TLS Issuing CA 06
https://crt.sh/?id=2841732837

The error was repeated by the same PKI operator across the CAs signed that day.

We should be done with our review of the prior years and provide a final update by the beginning of next week.

On 7) for the remediation plan, I can update that a new separate naming document that will only be used for SubCAs or Intermediates was created, and we are walking through how to complete this form next week with PKI Ops personnel. I have attached a sample of that template for externally operated subCAs to this bug.

Comment 9

[Brenda Bernal]

Attachment: external_Public_CA Naming Doc_sample.pdf

Comment 10

[Brenda Bernal]

Are there any further questions on the information supplied with Comments 8 & 9 above? We are now in full use of the new separate naming document for SubCAs with the previously attached file.

Comment 11

[Ben Wilson]

I think that this can be closed, which I'll do on or after 31-July-2020 unless additional issues or questions are presented.

Comment 12

[Ryan Sleevi]

I remain concerned about answers like Comment #8 ("We weren't done, but we provided an incident report claiming to provide the complete details"), and want to make sure DigiCert improves the quality and correctness of its incident reports going forward. This has been a repeat issue for a number of CAs, and while I value transparency, it does not take much effort to acknowledge whether reports are still in progress and, if so, the exact timeline for when that report will be completed (as required by https://wiki.mozilla.org/CA/Responding_To_An_Incident )

I support closing this, but remain concerned with DigiCert's approach to compliance and incident reports that has persisted for several years now, across a wide variety of issues.