1639805 - Sectigo: Failure to revoke key-compromised certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[mpalmer]

Steps to reproduce:

Between 2020-05-07 10:31:06 and 2020-05-09 19:39:19 (all times UTC), three certificate problem reports were sent to [e-mail], in case stating that a private key had been compromised, and requesting revocation of all certificates issued by Sectigo using the specified SPKI. The URL of a CSR attesting to the compromise of the private key, signed by the compromised private key, was provided. The sent time, SPKI, and receiving MX and IP address in each case are given below.

2020-05-07 10:31:06 8bf37942dc1fd6768fa0f00d42da6a015689bac50b0d2872392ecac7b7b05493 sectigo-com.mail.protection.outlook.com (104.47.46.36)
2020-05-07 10:36:50 14cf73729090f6755a9bdf17ad3bc3224ca6d2939e6d9bb29ec96c8f8f74cd36 sectigo-com.mail.protection.outlook.com (104.47.46.36)
2020-05-09 19:39:19 aa17f91507a2018dd3713aad9fe754f262490c4e7ecc242d86dbf4c94511a43e sectigo-com.mail.protection.outlook.com (104.47.44.36)

Actual results:

In each case, as of the time of writing, all valid, signed OCSP responses received for the impacted certificates show the certificate status as being valid. This includes certificates which are, at the time of writing, revoked -- OCSP responses with a produced at time of 2020-05-18 20:05:39 showed the certificate status as valid.

Expected results:

All certificates to have been revoked within 24 hours of the problem report being received.

Comment 1

[mpalmer]

Update: none of the certificates using the above public keys have been revoked (although two have expired).

Comment 2

[Ryan Sleevi]

Rich, Rob: Is there anyone at Sectigo who still pays attention to the expectations of being a publicly trusted CA? Can you make sure they see this? I’m not sure if we should be expecting Sectigo to announce they’ve left the CA business, which is usually what CAs end up doing when incidents systematically go ignored.

Comment 3

[Robin Alden]

Matt, Ryan,
First, sorry for the slow response.

In each case our customer service staff picked up the reports and validated them and sent a notification that the certificate would be revoked, but then the certificates were not revoked. There looks to be a broken process here so we will review all key compromise reports to identify which ones have not been actioned. It's partially broken, anyway, since some compromise reports are being actioned. I will report back here with the result of that review.

As Matt put in his problem reports to us, the certificates over those keys may be seen at:
https://crt.sh/?spkisha256=8bf37942dc1fd6768fa0f00d42da6a015689bac50b0d2872392ecac7b7b05493
https://crt.sh/?spkisha256=14cf73729090f6755a9bdf17ad3bc3224ca6d2939e6d9bb29ec96c8f8f74cd36
https://crt.sh/?spkisha256=aa17f91507a2018dd3713aad9fe754f262490c4e7ecc242d86dbf4c94511a43e

The first of those has now expired, but the key has been added to our list of compromised keys.
The other two have been revoked today with the reason showing key compromise.

Although we are not yet at a point where we automatically block issuance of further certificates using keys known to be compromised, we do have scripts being run manually, imminently to be automated, that revoke all certificates using keys marked as compromised. This script has the effect of revoking newly issued certificates with pre-compromised keys within 24 hours of issuance.

We have further code in development that will block the issuance of certificates over known-compromised keys.

This final statement does not relate to the particular compromise reports here, but we would be grateful to receive future reports of compromised keys through the automated mechanisms we have provided for this. Our recently updated CPS, available under https://sectigo.com/legal, includes the details of these, but for convenience I include a snippet from that document here.

• Sectigo also operates an automated revocation portal at https://secure.sectigo.com/products/RevocationPortal where Subscribers/Domain owners may revoke their certificates, or the public may report and revoke certificates for which the private key has been compromised.
• The public may also report and revoke certificates for which the private key has been compromised by using the ACME revokeCert method at this endpoint:
ACME Directory: https://acme.sectigo.com/v2/keyCompromise
revokeCert API: https://acme.sectigo.com/v2/keyCompromise/revokeCert

Comment 4

[Ben Wilson]

(In reply to Robin Alden from comment #3)

I will report back here with the result of that review.

Are there any updates you can provide here on that review of the revocation process? Thanks.

Comment 5

[Ryan Sleevi]

Nick: I'm hoping that, within the next 7 days, Sectigo and it's management will review every single open Sectigo issue, ensure that the requested information is present, and update Bug 1563579 with a comprehensive plan to address the complete and total systemic failure by Sectigo here to comply with expectations.

Comment 6

[Nick France]

Ryan: Yes, that process of reviewing and gathering information for responses is already underway, with updates being prepared for all open issues including 1563579.

Comment 7

[Rich Smith]

Robin has assigned the task of responding to this bug over to me.
In comment 3 Robin said, "...we will review all key compromise reports to identify which ones have not been actioned."
That review is complete and I will post the full results in the next day or two. I'm just running through and getting the crt.sh IDs, cross checking the list to see if any of the certs relate to other bugs, etc.

Comment 8

[Rich Smith]

We found 3 other certificates for which reports of key compromise had been received but the certificates had not been revoked. Those are:

https://crt.sh/?id=2445497992

This certificate is referenced in bug 1639518. It was set to revoked in our system so that any future certificates which might use the same key will get picked up by the script that checks for compromised keys and thereby also get revoked, however it was already expired by the time of this review, so no revocation information was ever published.

and

https://crt.sh/?id=2859493895
https://crt.sh/?id=2836926068

For which I can find no previous reference in any bugs. These certificates were revoked as soon as they were found as a result of this review.

Previous to this incident our customer service reps minding this queue had been notifying the customer of the key compromise and attempting to wait until the customer acknowledged, and possibly replaced the certificate with a new key, prior to revoking. This was found to be a significant factor to some of these ultimately being dropped. The process has been changed such that while the reps do still notify the customer, they do not await a response prior to revoking.

Comment 9

[Rob Stradling]

We have no further update.

Comment 10

[Ryan Sleevi]

Will there be an incident report, per https://wiki.mozilla.org/CA/Responding_To_An_Incident ?

Comment 11

[Rob Stradling]

Hi Ryan. Rich has been away this week. I'll ask him to prepare an incident report next week.

Comment 12

[Rich Smith]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   This bug as well as bug #1639804 alerted us to a problem with our front line reps responding to and actioning key compromise reports through our problem reporting email address.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   On June 1 the 3 certificates reported on this bug were revoked
   On June 2 an extensive review was performed of all other received problem reports received during the time period in which it had been identified that staffing levels may have been a problem. The 3 additional certificates mentioned above were found as a result of that review.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   Several certificates, as outlined above, were found to have not been revoked within the time required for which the key has been compromised.
5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   See above comments from Robin Alden and Rich Smith.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   Checking the certificate problem reporting queue at weekends was an issue in May.
   As Inigo stated on Bug 1639804, "We were struggling due to some external issues that affected our capacity to check out the abuse queue within the appropriate timeline." Additionally our reps were notifying customers prior to revocation (good) but also waiting for the customer to respond before actually revoking (not so good, for various reasons). This delay in taking action caused some reports to go unactioned because the rep would then go on to other tasks and forget to go back.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
   a) End of May we added one more person to check the queue and since then we have been much better catching the reports during non-working hours.
   There is no good way to catch these specific reports because it is always a human that need to check and find, so we´re training and assigning additional people to help check and manage for abuse reports and have some backups.
   b) We’ve also increased the oversight of the problem reporting queue by having the emails which come in also forward to select members of senior staff to ensure that the appropriate action is taken in a timely fashion.
   c) While our reps continue to notify customers of key compromise prior to revoking the certificate, they no longer await customer response before revoking.
   d) We continue to encourage the public to use the external facing tools we’ve released for reporting key compromise and revoking certificates:
   • Sectigo also operates an automated revocation portal at https://secure.sectigo.com/products/RevocationPortal
   where Subscribers/Domain owners may revoke their certificates, or the public may report and revoke certificates for which the private key has been compromised.
   • The public may also report and revoke certificates for which the private key has been compromised by using the ACME revokeCert method at this endpoint:
   ACME Directory: https://acme.sectigo.com/v2/keyCompromise
   revokeCert API: https://acme.sectigo.com/v2/keyCompromise/revokeCert
   e) We’ve developed better internal tools for our front line reps to confirm and action key compromise reports submitted via the problem reporting email address and will continue to monitor and improve these internal and external tools to automate as much as possible both for the public and for our own customer service staff.

Comment 13

[Rich Smith]

We have no further updates at this time. We feel that we have taken appropriate action to address the root causes of this bug. We will continue to monitor and make improvements as warranted. Ben, if there are no further questions or comments can this bug be closed?

Comment 14

[Ben Wilson]

I intend to close this on 9-Sept-2020 unless there are further questions, concerns or issues to be raised and discussed.

Comment 15

[Rob Stradling]

We have no further update.