1421820 - Microsoft DSRE PKI: Microsoft shares wildcard certificates among cloud instances

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Hanno Boeck]

I was informed about a problem with Microsoft involving a certificate whose private key is essentially public. This is not my own research, I learned about this from Matthias Gliwka.

Microsoft offers a cloud-based service for Dynamics 365 for Operations. The cloud instances have host names of the scheme [customer].operations.dynamics.com, there's also a "sandbox" instance that is intended for testing and uses [customer].sandbox.operations.dynamics.com. Matthias Gliwka only tested this on the sandbox instance, however there is no reason to believe that the same problem doesn't exist on the production instance as well.

All customer instances share the same wildcard certificate and private key. This is the affected certificate:
https://crt.sh/?id=51380275
The certificate of the production instance is here:
https://crt.sh/?id=93343221

These cloud instances allow customers to log in. Therefore customers are able to extract the private key from them (mimikatz can be used for this). Thus this "private keys" isn't private, every user of that service can extract it.

According to the Baseline Requirements certificates with compromised keys shall be revoked within 24 hours. According to Gliwka's description he has been in contact with Microsoft since August. In first replies Microsoft denied that this is a security problem, later they were unable to find the case again and later attempts by Gliwka were unanswered. The certificate is still valid.

He was able to show me the private key, so I have no reason to believe any of this is not true.

I intend to make this public soon, but I wanted to make sure Mozilla is aware, as this obviously affects the root program. I'll CC some people who commonly deal with CA issues.

Comment 1

[Hanno Boeck]

Adding Digicert to cc as the Microsoft CA is signed by a digicert-owned root.

Comment 2

[Kathleen Wilson]

Attachment: tls_leak_ms.pdf

Comment 3

[Kathleen Wilson]

Also note: https://bugzilla.mozilla.org/show_bug.cgi?id=1398269#c17
(comment was in wrong bug)
which says:
Matthias Gliwka agreed that I share the document where he describes all the details here. He plans to publish this as a blog post once this is sorted out.

Comment 4

[Kathleen Wilson]

Wayne, Would you please dive into the details of this bug and recommend Mozilla action?
i.e. should we add the cert to OneCRL? If yes, what would be the expected result? How big of a compatibility problem will this be? 

Karina, Would you please have Gordon create a Bugzilla account, so you can add him to the CC of this bug?
https://bugzilla.mozilla.org/createaccount.cgi
It will be very helpful to have Microsoft weigh in on this bug.

Thanks!

Comment 5

[Gervase Markham [:gerv]]

I don't think we should add this to OneCRL without giving Microsoft a day or two to respond; I suspect the compat impact will be fairly large. I also suspect that fixing it will require them to rearchitect their service somewhat, perhaps by inserting HTTPS terminators in front of all the customer instances, and that will take time. (As would switching to per-instance certificates.) It's not good that they've been brushing this off so far, but we also need to acknowledge that the fix is not trivial.

Gerv

Comment 6

[Gordon Bock]

Thank you for identifying this.  Microsoft is investigating this issue.

Comment 7

[Jeremy Rowley]

Sorry - just saw this bug. Let me know what we need to do with the issuing CA.

Comment 8

[Jeremy Rowley]

Err... the end entity cert. This wasn't reported to us as a certificate problem report, so it hasn't triggered the 24 hours yet.

Comment 9

[Kathleen Wilson]

(In reply to Jeremy Rowley from comment #8)
> Err... the end entity cert. This wasn't reported to us as a certificate
> problem report, so it hasn't triggered the 24 hours yet.

I understand the intent of the BR 24-hour rule, but we also need to weigh in the consequences of revoking such certificates, etc. So let's give Microsoft some time to respond with their plan.

Thanks!

Comment 10

[Kathleen Wilson]

Quick update: Microsoft representatives have informed us via email that they are actively engaged on this issue, and have requested that the attached report not be published yet. They plan to provide further information by Monday. 

Thanks!

Comment 11

[Kathleen Wilson]

Hanno, would you and Matthias Gliwka please contact Gordon Bock and Mike Reilly (CC list above) directly?

As for this particular Bugzilla Bug, I would prefer to wait for Microsoft's own statement and recommendation to root store operators regarding things like OneCRL, and then handle OneCRL updates in a public-facing bug.

Thanks!

Comment 12

[Kathleen Wilson]

Closing as resolved fixed, per email from Microsoft. (though I will probably create a separate bug to add the two wildcard certs to OneCRL).

Please keep this bug hidden. Microsoft will be handling all public-facing communication about this.

Thanks!

Comment 13

[Hanno Boeck]

I had mailed microsoft the following questions today (no answers yet), I think particularly the question about the *.trial.operations.dynamics.com certificate is relevant here.


* Based on claims made by your press department and made by colleagues
  of you in a mail to Matthias Gliwka it wasn't possible to extract the
  key from the production instance, as there is no rdp possible.
  However he informed me that there is a possibility to deploy custom
  software extensions on those systems and thus execute custom code. I
  am not familiar with these systems, but this sounds like one could
  just execute a custom extension that extracts the private key and
  leaks it somehow.
* I see three wildcard certificates for operations.dynamics.com:
  https://crt.sh/?q=%25.operations.dynamics.com
  Two of them have been revoked, but there's a third one for
  *.trial.operations.dynamics.com - can you explain to me where this is
  used, what the difference between "trial" and "sandbox" is and
  whether the "trial" certificate was used on any hosts where the same
  problem might exist? (And thus it should be revoked as well.)

Comment 14

[Kathleen Wilson]

I filed Bug #1423400 to add the two entries to OneCRL for.
https://crt.sh/?id=51380275
https://crt.sh/?id=93343221

Thanks!

Comment 15

[Gervase Markham [:gerv]]

Can someone link to some of the public discussion or announcements about this?

Kathleen: what did the email from Microsoft actually say?

Gerv

Comment 16

[Hanno Boeck]

I haven't published anything yet, but will probably do tomorrow, I'll post the link asap.

Comment 17

[Kathleen Wilson]

(In reply to Gervase Markham [:gerv] from comment #15)
> 
> Kathleen: what did the email from Microsoft actually say?
> 

Here's snippets:
~~
>> The two certificates have been revoked on our side.  You can now revoke on OneCRL.

>> Please DO NOT REVOKE *.trial.operations.dynamics.com ... received confirmation that this certificate is in use and there is no risk of exposure for this private key.
~~

I do not plan to say anything public (other than the CA asked to add those two certs to OneCRL) until Hanno or Microsoft make public statements about it.

Comment 18

[Hanno Boeck]

Public:
https://www.golem.de/news/microsoft-dynamics-365-wildcard-certificate-with-a-private-key-for-everyone-1712-131544.html
https://medium.com/matthias-gliwka/microsoft-leaks-tls-private-key-for-cloud-erp-product-10b56f7d648

From my side I think this bug can also be made public now.

Comment 19

[Gervase Markham [:gerv]]

Opening bug at request of reporter.

Gerv

Comment 20

[Wayne Thayer]

I filed bug 1424305 requesting an incident report from DigiCert.