1624504 - QuoVadis: Failure to revoke certificates with compromised private keys

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Stephen Davidson]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

QuoVadis received an email from Matt Palmer overnight on 3/20/2020 at 00:05 AST as follows:

-----Original Message-----
From: Matt Palmer <[e-mail]>
Sent: Friday, March 20, 2020 12:05 AM
To: QV - QuoVadis Compliance
Subject: Certificate Problem Report: multiple certificates with compromised private keys
Certificates issued by your CA have been found to be using private keys which have been publicly disclosed. The list of SPKI fingerprints of the keys for these certificates are:
6ca25b96d613ed380d4285a450b1737346ab167bb32dcd8289f4bb9445e4521f
051c7cd46144458d508021dd721460b718c8bf5b3a4a20cfb2a7d9dd1f25470e
1737e8265beefa81310deaacb8e957231fc412f1babb5c3b4fe59b0dcae09cf7
1e71edcb5c1c93180ce063cb50e11ea5867638ec8af9f0be39c8da618e41dd3a
37c5f57216ce204702c60587e0f0b6c821f9ebb145129a780512f444e48d79fe
4a92dfd0de92af47e2a0b90f50854316bb07cc58ceb5dd82f817de51f7f8851d
5eef28fbfe1e59b19d4b7db546d95701ea618aef0ed355a2b7f41ebaf5bd07d7
7cf87d510b4bd8899ea9c48956e8fdc6893778c613148fd174d7d1a030797dc1
8317e90c1fcc6398b4530554d298a4d9268ca115806e231a1b7e9744930ea535
84adaec99f45eb4412fab4b4553670c56e920d998d7bc7611193568f88a000a0
91c1befee55e20f7d972aef5305083cf4ee0c9c951e17516fcfe51a372450839
9f6d02a745bfcc796d3270ed25f5a01654d503f464f50d2274c297b58567193e
aad925a8b38626c0b16eeb6e7d4aba571bd4f0d0232a34b9315c1034920b451e
bec0799a6122e65be89ca6f992b62ef71c3d1c566fd7d55438d88a75d685436f
bf7ee7afb5a618e9c67ef188f6ccb9f9a1d909b0e3958ad860b32d4db9cd181d
c39395e26c5401740fc4d3aef6feab97dda7509be7608b4029d7ea08b33c46d5
cc10498794c701df174e7cdeaeaa15c891f9b8ec8babe58138ae3fffc3b720cc
d03309be836b28243fa9a3a97d0e8d85ebe9e4a93e21f8842e261550f042f78d
da443e4cadd5109e564676ade6140a3f18ce36550be207006d672e808f9fdf17
dbc3dc447144ad3d9da596527fa8dfb86e6b4d7dc2919ea5b45c91679e250619
e5582b0b3dc60008bacc27e484934439c1a4ba12b328ed45e9628772ff40f1b8
ea87fa7963c0c97c98a6595da74ce650325eeb8d8d44c1aabf2239db376b3412
ebc35ec62cb941ada24fd53376c80bb76650aeab27e969174978e848adb5a776
You can obtain a list of the certificates which use each of these SPKI fingerprints by visiting: https://crt.sh/?spkisha256=<fingerprint> A PKCS#10 DER-format CSR, signed by the private key and attesting that the private key has been compromised, can be retrieved from: https://v1.pwnedkeys.com/<fingerprint>.csr
Please revoke all affected certificates within 24 hours, as per the Baseline Requirements.

• Matt

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

As the email was sent in the early morning hours, I did not receive it until the next morning. An acknowledgement was made to Matt on Saturday 3/20/2020 at 10:05 AST stating "We acknowledge receipt of your problem report relating to the below certificates this morning. We will investigate the certificates and, if they are found to be noncompliant, will revoke the certificates within the stipulated 24 hours. We will update you with the outcome at that time."

SHA-256 fingerprints have been the lingua franca in certificate problem reports, and the non-routine SPKI format used by Matt required some time using external systems to manually identify the certificates as standard admin views into the QuoVadis certificate management system do not include SPKI (see note 7).

As the identified certificates were held by universities who are currently teaching remotely, an alert was sent to the customers advising them to replace certificates urgently using new key pairs as their previous private keys had been reported compromised.

The certificates were revoked late on Saturday evening, and an email was sent to Matt on Sunday 3/21/2020 at 00:07 AM AST stating "These certificates have been revoked. In the process, I identified some additional certificates which share certain of these Public Keys. I require additional assistance to research this thoroughly, which will take place tomorrow. Additional certificates we identify will be revoked in a further 24 hr cycle."

In triggering the revocation, I noticed several certificates beyond those initially reported by Matt shared the same keys and added them to the revocation. I also identified that a customer had reissued some of the affected certificates after my alert but before the revocation, reusing the compromised keys.

As noted in my email to Matt, I was unable to search by SPKI within our systems and required additional QuoVadis assistance to make sure we had found all the certificates. I did not have confidence in using the external CT monitors as I was able to find some of the newly customer-reissued certs in our systems but not yet in crt.sh and censys.

The search in our systems was completed later in the morning of Sunday identifying two additional certificates; the client was informed, and revocation was scheduled for ~22:00 AST on Saturday 3/21/2020.

Before that could occur, Matt filed the following (identifying the two certs that were in the queue for revocation, along with a stray precert).

-----Original Message-----
From: Matt Palmer <[e-mail]>
Sent: Saturday, March 21, 2020 18:23 PM
Subject: QuoVadis: Failure to revoke key-compromised certificates within 24 hours

Three certificates were reported as having private keys which had
been publicly disclosed, by e-mailing [e-mail] at
2020-03-20 03:05:14 UTC. E-mail was received by a QuoVadis server at
2020-03-20 03:05:18 UTC. As of 2020-03-22 05:17:37, OCSP still shows all of
these certificates as being "Good".
The unrevoked certificates are:
https://crt.sh/?id=2605016622
https://crt.sh/?id=1757153116
https://crt.sh/?id=1432019792
Interestingly, at least one other certificate using the same private key as
each of the above certificates, and also issued by QuoVadis, are now showing
as revoked, suggesting that (a) QuoVadis did indeed consider the private
keys as compromised, and (b) there are no caching or delayed publishing
issues at play here.

• Matt

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Not applicable.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

204F45147A16D8951E7B9C05AA88FABBC7ACE7C5
7404E6DF14E7F7B860036CBD3E661C5FE6ECADA8
855721544A2B7B1ACD0F1F3A56167F5C9B4224DA
DA6C33A4762DADD9AEC39902C25B57E0991D726D
56F390E32B9B5B1BC78CC4D25CC029348683D06F
6BACF71A9E7F57620BBDEF36DD8DDB02DD0E1DFE
651AF4E879F8406FB09213AA1504C728DF527196
50B32560A8672E84DA308268D21E9AA95A114E70
DDD85045F5AB407FC027A9B63AFB9E1308CD23D1
1B856A0D5B4110858BBC099FD036446513AEE3B4
95D54A4EAAD72A89E190C23C71A110538B18D8F0
7CF7DB7FF38571B76EF12391DC21072B8170FA39
1CB42F872F80080ED768CAD9F4B729AF677E86E8
BF3EC937E041BA0D6C53E836C30742C5672F0431
F12734A9F9A299DBF39824A1936E734754CEF85E
A9CF4586AA6A97AD586538281E71F3CA57406B2C
448EB677C3B932D11A75010F809011DC266AF18E
164AEA2BA76429E6E2CB1C7A6EE9F1E566D19AFF
AF9B4FEE7DC08567F3B090AACA71804689A9FEE2
684ADA8FB4DDE4A7468166A59668953DDE86348C
E72C1E1AB4159FC33AC92F1D4AD1A765F6CA8B03
10A6A77C4D55C1F62F07BE842D201B5F20DAF587
76AC85162920ED250D6797F22366909E7B12B756
EB2F30E7C94283632F3961776290BDED6FB3B2F5
98AAF2A6BCA84FD92AA763176A6E14D3094FCE7F
D6C579994D6F40832E79D979B10078CD22F522BA
9B7DC3B9187A4718C533C688C5D182BC893DD255

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It would be desirable if the means of identifying certificates and compromised keys could be standardized across CAs via the Baseline Requirements. This would allow CAs to develop appropriate tooling, increase automation, and reduce turnaround when revocations are required.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

A change will be made to improve searching using SPKI for administrators in our Certificate Management System. This is expected to be in place within 15-20 days.

Comment 1

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #0)

As the email was sent in the early morning hours, I did not receive it until the next morning.

I thought, as part of Bug 1590171, that DigiCert's Compliance Team was involved in QuoVadis reports going forward. This seemed to be very clear from Jeremy Rowley's remarks. Certainly, while the initial response appears to be within that 24h window, I mention it, because it seems like had the clock to begin investigating happened earlier, or had those mitigations mentioned in that prior bug been implemented, it seems like this incident would and could have been avoided.

In triggering the revocation, I noticed several certificates beyond those initially reported by Matt shared the same keys and added them to the revocation.

I'm having trouble understanding this. Matt's e-mail doesn't list certificates, but public key hashes and their corresponding evidence of private key compromise. Can you elaborate?

For example, I could see a possible interpretation being "QuoVadis examined the certificates disclosed in CT, but also determined there were certificates not disclosed via CT", but that doesn't seem to fit the discussion from the thread

I also identified that a customer had reissued some of the affected certificates after my alert but before the revocation, reusing the compromised keys.

This seems like an incredibly important opportunity for improvement that isn't captured in your proposed remediation plan. It seems that blocking new issuance should be a key step in your revocation playbook that happens prior to any notification or revocation.

As noted in my email to Matt, I was unable to search by SPKI within our systems and required additional QuoVadis assistance to make sure we had found all the certificates. I did not have confidence in using the external CT monitors as I was able to find some of the newly customer-reissued certs in our systems but not yet in crt.sh and censys.

I'm not sure I understand this. My understanding is certificates that could have been found this way were ignored? While I can understand not wanting to take a complete view until you've examined your own system, as systems like crt.sh/censys have delays in ingestion and CT Logs have up until their MMD to publish, meaning there can be 24 - 48 hour delays in the worst case, it does seem like it caused a missed window in responding timely when the data was available.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It would be desirable if the means of identifying certificates and compromised keys could be standardized across CAs via the Baseline Requirements. This would allow CAs to develop appropriate tooling, increase automation, and reduce turnaround when revocations are required.

I'm not sure what this means at all. This seems like it was entirely within QuoVadis' control, and with QuoVadis' system, so it's unclear where, how, or why the Baseline Requirements would be helpful. This seems no different than a CA that may need to scan for problematic O values or problematic Jurisdiction of Incorporation information.

If you mean the reporters should identify the relevant certificates, it's unclear why that should be reasonable. Indeed, this has only recently become a possibility, given Certificate Transparency being required, and prior to CT, the most a reporter could ever do for private key compromise was report the associated public key. That's why it's unclear why the system wasn't already capable.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

A change will be made to improve searching using SPKI for administrators in our Certificate Management System. This is expected to be in place within 15-20 days.

As mentioned above, there seem several missed opportunities here for closer examination of both root causes and mitigations.

• QuoVadis does not readily have the capability to quickly examine its own system for issued certificates
• QuoVadis did not make ready use of existing sources of QuoVadis-issued certificates (such as CT or Censys)
• QuoVadis does not have a process in place that defines a sequence of steps for revocation to prevent issues, namely:
  • QuoVadis does not block compromised keys prior to notifying the customer and/or revoking certificates

As mentioned earlier, had Bug 1590171 been implemented, it would appear this issue could have been prevented. While I was not thrilled with the level of incident response on that bug, I had hoped QuoVadis would be taking steps to improve the introspection and analysis for these issues.

Comment 2

[Jeremy Rowley]

Hmm - based on this incident report and what Ryan mentioned, I think we have some organization issues still with the integration between Digicert and Quovadis. Let me work internally to figure out why no one picked this up before Stephen on Quovadis time. Support people for Quovadis should have been able to handle the standard revocation without Stephen being directly involved.

For clarity, the compliance team was on the original email from Matt to Quovadis. However, we didn't see a response from Stephen to Matt. This didn't raise red flags for us because we usually don't see the response (at Digicert the support team takes care of revocation, not the compliance team). We implemented the process where we see the incidents at Quovadis and the revocation requests coming in, but the compliance team doesn't have access to the Quovadis data.

Copying from the other bug:
As part of the integration of QuoVadis into DigiCert, the QuoVadis compliance team now reports to DigiCert. In addition, an additional experienced internal auditor has been hired by DigiCert whose role includes the 3% review of QuoVadis TLS issuance.

• This was done. We have a 3% auditor that works with the Digicert compliance team. We also merged compliance teams. Revocations are handled by support as part of their routine operations. I need to dig into why support didn't handle the Quovadis revocations.

Previously TLS validation was performed by separate national subsidiaries of QuoVadis for their respective client bases. Since the acquisition by DigiCert, QuoVadis validation has been centralized into one team with a group leader, and reports into the larger DigiCert validation group. Three highly experienced validation staff from DigiCert have been seconded to the group.

• Validation is consolidated. I thought support was also consolidated so I'm not sure why there was a gap. Will dig into this and get back to you.

Short term, updated training has been provided to QuoVadis RAs relating to EV JOI and businessCategory fields.

• This was done.

Longer term, QuoVadis TLS issuance will move to DigiCert’s platforms, benefiting from DigiCert’s significant investment in automation and validation tools. A roadmap will be provided at a later date.

• We're working on a roadmap now and plan to start the integration in Q2 with a goal of having all migration from the Quovadis TLS system to DigIert's issuance system in early 2021.

Comment 3

[Ryan Sleevi]

(In reply to Jeremy Rowley from comment #2)

Hmm - based on this incident report and what Ryan mentioned, I think we have some organization issues still with the integration between Digicert and Quovadis. Let me work internally to figure out why no one picked this up before Stephen on Quovadis time. Support people for Quovadis should have been able to handle the standard revocation without Stephen being directly involved.

Jeremy: Do you have an update here? Are y'all still doing your weekly syncs to make sure every bug gets updated weekly? It sounds like there are still some process breakdowns happening here at DigiCert (and acquisitions), and that's more than a little concerning, especially when they've previously been called out.

Comment 4

[Jeremy Rowley]

Yes - we do weekly syncs, and I've been assigned to answer the bug.

I thought, as part of Bug 1590171, that DigiCert's Compliance Team was involved in QuoVadis reports going forward. This seemed to be very clear from Jeremy Rowley's remarks. Certainly, while the initial response appears to be within that 24h window, I mention it, because it seems like had the clock to begin investigating happened earlier, or had those mitigations mentioned in that prior bug been implemented, it seems like this incident would and could have been avoided.

The QuoVadis compliance team is assigned to compliance issues. However, revocations are handled by support, not compliance. What we found is that the current path through support only applied to DigiCert and not QuoVadis - ie we combined compliance and validation, but not support. We should have done so, but when we split compliance and support into two separate teams we ended up having support for Quovadis different than support for DigiCert. It made sense at the time since the QuoVadis systems are still through QuoVadis (no overlap with Digicert). What didn't make sense is that revocation processing is separate. We've moved all revocation for key compromise to Digicert support.

I'm having trouble understanding this. Matt's e-mail doesn't list certificates, but public key hashes and their corresponding evidence of private key compromise. Can you elaborate?

QuoVadis did not have a way to search by SPKI/key in their system. Today, they can only search by serial number or fingerprint. They're working on adding better searching functionality in the next two weeks (the teams are working on it now).

For example, I could see a possible interpretation being "QuoVadis examined the certificates disclosed in CT, but also determined there were certificates not disclosed via CT", but that doesn't seem to fit the discussion from the thread

That is part of the Support process on the DigiCert side. Moving revocation processing to DigiCert Support will address this.

This seems like an incredibly important opportunity for improvement that isn't captured in your proposed remediation plan. It seems that blocking new issuance should be a key step in your revocation playbook that happens prior to any notification or revocation.

One note on this is that we don’t have a blacklist that is supported across the organizations. DigiCert v. QuoVadis v. legacy Symantec all have their own way of dealing with revocations. Both DigiCert and Symantec are the same systems (for TLS) now so that should be mostly uniform at this point. We still don’t have a way for QuoVadis and DigiCert to check the same blacklist. As part of the remediation on the DigiCert bug, I mentioned building a key blacklist. We’re designing this as a service that both QuoVadis and DigiCert can tie into. It’ll be great because it’ll ensure a report to QuoVadis will also be implemented at Digicert and vice versa, and a shared service will help make the migration from QuoVadis to DigiCert easier.

I'm not sure I understand this. My understanding is certificates that could have been found this way were ignored? While I can understand not wanting to take a complete view until you've examined your own system, as systems like crt.sh/censys have delays in ingestion and CT Logs have up until their MMD to publish, meaning there can be 24 - 48 hour delays in the worst case, it does seem like it caused a missed window in responding timely when the data was available.

Because of the inability of searching by SPKI/key in their certificate system, Quovadis didn't find it all. We have access to Censys on the DigiCert side. With the combination of support, Quovadis will have better CT log scanning going forward (for compromised keys).

I'm not sure what this means at all. This seems like it was entirely within QuoVadis' control, and with QuoVadis' system, so it's unclear where, how, or why the Baseline Requirements would be helpful. This seems no different than a CA that may need to scan for problematic O values or problematic Jurisdiction of Incorporation information.

Stephen was alluding to the way Matt reports key compromise. Really though we want to accept various revocation requests as they come in.

QuoVadis does not readily have the capability to quickly examine its own system for issued certificates

This is sort of true in that they can't search by public key. They can search by serial number and fingerprint, among many other options. They can't search by SPKI/key. Their engineering team is working on that.

QuoVadis did not make ready use of existing sources of QuoVadis-issued certificates (such as CT or Censys)

Part of the standard procedure is to search for CT. Moving support to Digicert support will solve this.

QuoVadis does not have a process in place that defines a sequence of steps for revocation to prevent issues, namely: QuoVadis does not block compromised keys prior to notifying the customer and/or revoking certificates

This is true at both DigiCert and at QuoVadis. Neither company has the ability to blacklist keys before revocation. We’ve reached out to PrimeKey to see if/how it can be added as a capability on EJBCA. In the meantime, we’re specc’ing out the service per the DigiCert bug report. Then we’ll figure out (how) QuoVadis can use that service for timely research and action.

As mentioned earlier, had Bug 1590171 been implemented, it would appear this issue could have been prevented. While I was not thrilled with the level of incident response on that bug, I had hoped QuoVadis would be taking steps to improve the introspection and analysis for these issues.

Bug 1590171 was implemented with the exception that I thought support had also moved over. We’re rectifying that now and moving all revocations through DigiCert.

The short-term plan for QuoVadis is:

1. Make it so we can search by SPKI/key (next two weeks)
2. Block CSRs with a SPKI that matches a known compromised key
3. Automatically block SPKIs if the reason for the revocation is key compromise

The long term plan is still to migrate Quovadis to Digicert systems.

Comment 5

[Ryan Sleevi]

Thanks for the update, Jeremy.

I'm holding off setting a next update, so that you can update us weekly to make sure you're progressing on SPKI/key searching.

Comment 6

[Jeremy Rowley]

The SPKI searching is live on the Quovadis platform. Key blacklisting across all platforms is planned with an estimate of it being done end of May. The project will carry over a bit into June while we configure each platform to call into it.

Comment 7

[Stephen Davidson]

QuoVadis confirms that:

• SPKI searching is available for administrators in our Certificate Management System,
• CSRs including keys matching those previously revoked/keyCompromise by QuoVadis CAs are rejected, and
• QuoVadis will later adopt the DigiCert blocklist described by Jeremy in comment 6 when it becomes available.

We request this bug be closed.

Comment 8

[Wayne Thayer]

This seems like a good place to track the implementation of the DigiCert blocklist described above, so I'm setting next update to mid-June.

Comment 9

[Jeremy Rowley]

Thanks Wayne. Looking at this one, we probably can't close it until the DigiCert blocklist is finished. Although Quovadis did implement a better way to detect and revoke certificates, Quovadis has the same issue everyone else has - you can still get a cert with a compromised key in the period between revocation and when the search is complete. Stephen did report what Quovadis did, but there is still a risk of compromised key reuse until we finish the blacklist in #6 (eta is June).

Comment 10

[Jeremy Rowley]

I made an update on the other bug, but thought I should replicate it here. We're planning on launching the initial key blacklist checker on June 30. After that we will work on having Quovadis use this same blacklist checker. We've also kicked off a meeting on the tasks required to consolidate Quovadis systems into DigiCert. We're targeting the end of the year to have the systems ready for consolidation, with the goal of migrating Quovadis into DigiCert systems early 2021.

Comment 11

[Jeremy Rowley]

We launched the key blacklist checker for DigiCert, meaning Quovadis can't use it yet. Given the OCSP issue, we are going to re-evaluate the timelines for customer migration to DigiCert's systems. I'm thinking we do it this year as part of the remediation for the OCSP issue.

Comment 12

[Stephen Davidson]

As described in Comment 7, DigiCert has created a service allowing group Issuing CAs including QuoVadis to both check and contribute to a central blocklist of keyCompromise keys. This will be in full production at QuoVadis during the week of July 27.
This work also lays the groundwork for QuoVadis legacy CAs to check/contribute with the group regarding the use of previously CA-generated Key Pairs within Subscriber TLS certificates.

Comment 13

[Stephen Davidson]

On 29 July, the ability to check and contribute to DigiCert's central blocklist of keyCompromise keys was implemented in QuoVadis CAs.
Unless there are other questions, QuoVadis requests the closure of this bug.