1424305 - DigiCert: Microsoft: Incident report for Microsoft Dynamics incident

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Bug 1421820 documents a security incident with a certificate issued by a subordinate CA run by Microsoft and signed by DigiCert. A number of concerns arose that make this worthy of an incident report:

- The CA's problem reporting mechanism was unclear, at best.
- Action wasn't taken within 24 hours
- The CRL for the sub-CA wasn't functioning
- The revocation reason was marked as 'cessation of operation' despite evidence of key compromise

Requesting that DigiCert file a standard incident report.

Comment 1

[Kathleen Wilson]

Jeremy and Ben, DigiCert is responsible for providing the Incident Report to Mozilla by attaching it to this bug, because the problem was the handling of private key compromise by a subCA of DigiCert. 

https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Comment 2

[Jeremy Rowley]

Okay. I'm reaching out to Gordon to get a full report.  We never received a certificate problem report through the expected channel for this cert.

Comment 3

[Kathleen Wilson]

(In reply to Jeremy Rowley from comment #2)
> We never received a
> certificate problem report through the expected channel for this cert.

Does DigiCert require their subCAs to report such problems?

Comment 4

[Jeremy Rowley]

Sort of.

We require a report of all non-compliances. Our agreement with each customer requires the following for key compromise: "Customer will notify DigiCert, request revocation of a Certificate and its associated Private Key, cease using such Certificate and its associated Private Key, and remove the Certificate from all devices where it is installed if (i) any information in the Certificate is or becomes incorrect or inaccurate, (ii) there is any actual or suspected misuse or compromise of the Private Key associated with the Public included in the Certificate." 

For customers with issuing CAs, there is an obligation for them to follow the CPS and report compliance issues. 

As Wayne noted, the compliance issues were
1) The CA's problem reporting mechanism was unclear, at best. 
[Publication of instructions online is required under BR 4.9.3]
2) Action wasn't taken within 24 hours
[Required under 4.9.5]
3)  The CRL for the sub-CA wasn't functioning
[24x7 capabilities areq required under 4.10.2]


The last one "The revocation reason was marked as 'cessation of operation' despite evidence of key compromise" isn't a compliance issue. I'm not aware of a policy that requires specific reason codes or anything related to them. It's definitely a good practice. Perhaps something we bring up at the CAB Forum? 

I actually don't mind getting certificate problem reports directly for any Sub CA. We generally have better contacts than what you can find online, notwithstanding the BR requirement about a public reporting mechanism. If we get the certificate problem report directly, we can monitor the time it takes and work on a certificate problem report right away.  Plus, we went through the root store embedment process so any problem with a sub is a problem with DigiCert. 

I have a call with Microsoft tomorrow. I'll post an update shortly.

Comment 5

[Kathleen Wilson]

Jeremy, any update on this?

Comment 6

[Jeremy Rowley]

Here's what I received so far:

Digicert and the customer was made aware of the compromise as soon as it was verified. This incident was tied to risks associated with how a specific end-level certificates was used in its application environment. Issue validation and remediation have been completed along with the certificate revocation. This does not negatively reflect how the certificate was issued and managed.  Certificate information can be found at https://crt.sh/?q=*.sandbox.operations.dynamics.com  

I'm working on getting an update.

Comment 7

[Wayne Thayer]

Jeremy: when can we expect to see this incident report?

Comment 8

[Jeremy Rowley]

Today!

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.
a.	This issue arrived at Microsoft directly to an individual in one of the PKI operating teams via email from Mozilla root program on November 29th. The MSIT PKI group that manages the issuing CA was not made aware of the issue until December 8th after the certificate was already revoked.  

2.	A timeline of the actions your CA took in response.
a.	The responding team followed their internal practices and made DigiCert and the customer aware of the compromise on 11/30. 

3.	Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.
a.	The certificate in question was revoked on Dec 4th. The issue was a process handling issue of the private key and not a problem with the issuing CA.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
a.	The one cert is question was valid Not Before: Oct 29 20:09:06 2016 GMT and Not After : Apr 29 20:09:06 2018 GMT until revoked @ 2017-12-04  20:34:38 UTC.

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
a.	https://crt.sh/?id=51380275 

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
a.	This incident was tied to risks associated with how a specific end-level certificates were used in its application environment and does not negatively reflect how the certificate was issued or managed. 

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
a.	Both internal PKI groups will now use a common response mechanism using out internal Response Center

In Summary:
An issue was reported to one of Microsoft’s certificate services organizations on November 29th.  The reporting mechanism did not follow the path in the issuer statement as outlined in the CP/CPS.  The reporting team managed the event as they would through their own practice statement.  DigiCert was made aware of the compromise on 11/30 along with our customer.  Issue validation and remediation was completed on 12/4 after which the certificate was immediately revoked. This incident was tied to risks associated with how a specific end-level certificates were used in its application environment and does not negatively reflect how the certificate was issued or managed. These two managing team have now sync’d on using a common response method. Certificate information can be found at https://crt.sh/?q=*.sandbox.operations.dynamics.com

Comment 9

[Wayne Thayer]

From now on, if I send problem report to [e-mail], an investigation will begin within 24 hours - is that correct?

Comment 10

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 11

[Jeremy Rowley]

I've asked to make sure. You can also send it to [e-mail]. We're happy to process those requests there.

Comment 12

[Ryan Sleevi]

As an independently operated sub-CA (correct?), shouldn't they be capable of complying on their own?

Comment 13

[Jeremy Rowley]

Correct, and yes - they can comply on their own. But we're happy to help so I definitely don't mind if we get a notice as well as them.

Comment 14

[Wayne Thayer]

(In reply to Jeremy Rowley from comment #13)
> Correct, and yes - they can comply on their own. But we're happy to help so
> I definitely don't mind if we get a notice as well as them.

Jeremy:
As an independently audited sub-CA, Microsoft is bound by the BRs, including the requirement to respond to incident reports within 24 hours. Do you agree?

As the root CA in Mozilla's program that signed Microsoft's sub-CA, DigiCert is fully responsible for the compliance of Microsoft's sub-CA. Do you agree?

If yes to both, then I'm asking you to investigate and remediate a compliance issue with one of your sub-CAs. The fact that reporting the problem to DigiCert might have produced a better outcome isn't relevant. If you'd like to have Microsoft publish [e-mail] as their problem reporting mechanism, that would solve the problem for me.

Comment 15

[Jeremy Rowley]

1) As a publicly trusted CA, Microsoft is bound by the guidelines.
2) As a subordinate off DigiCert, DigiCert is responsible for the compliance.

Note that I'm not saying the email should have been sent or should in the future be sent to [e-mail], but I'm happy to take sub CA partner emails there as well. We don't get a copy of items sent to [e-mail].

I do agree that the [e-mail] suggestion isn't relevant to this particular issue and off-topic for the remediation discussion (consider it withdrawn). Still trying to confirm whether that is the email address they want to use for revocation.

Comment 16

[Jeremy Rowley]

MS confirmed the address is where you can send emails to initiate an investigation will begin within 24 hours.

Comment 17

[Wayne Thayer]

All action have been completed, marking this resolved.