1619359 - Sectigo: Failure to provide a preliminary report within 24 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Sleevi]

Matt Palmer reported on m.d.s.p. the following:

Between 26 Feb 2020 00:48:11 UTC and 26 Feb 2020 21:10:18 UTC, I sent three
Certificate Problem Reports to [e-mail], reporting that
certificates issued by then were using keys which have been compromised due
to being publicly disclosed. As of the time of writing, I have not received
a preliminary report of Sectigo's findings, as I believe is required by
section 4.9.5 of the Baseline Requirements.

In each case, I received an auto-acknowledgement e-mail containing a case
number, which indicates that Sectigo did, in fact, receive my problem
report.

Due to a mistake on my part, the evidence I provided to Sectigo was not
sufficient to verify that the key was in fact compromised, so I am not
claiming that Sectigo has fallen foul of BR s4.9.1.1. However, as BR s4.9.5
require a report to be provided within 24 hours, I still believe Sectigo
has an operational deficiency which requires investigation.

The times of the e-mails I sent, the Sectigo case number I received in
response, and the further responses I have received from Sectigo, if any,
are detailed below. All times are taken from the Date header of the
relevant e-mail, adjusted to UTC if required.

Case #00572387
https://crt.sh/?id=2455920199
Sent: 26 Feb 2020 00:48:11 +0000
Auto-ack: 26 Feb 2020 00:48:24 +0000

At 27 Feb 2020 19:15:10 +0000, I received an e-mail purporting to be from
Sectigo Security, quoting my initial report, and saying "we will look into
this right away". Note that even this response, which I do not consider
qualifies as a "preliminary report", was sent over 24 hours after the
initial problem report.

No further response has been received since then.

Case #00572465
https://crt.sh/?id=2413850414
Sent: 26 Feb 2020 05:07:34 +0000
Auto-ack: 26 Feb 2020 05:07:45 +0000

No further response has been received since the auto-acknowledgement.

Case #00573105
https://crt.sh/?id=683622319
Sent: Wed, 26 Feb 2020 21:10:18 +0000
Auto-ack: Wed, 26 Feb 2020 21:10:32 +0000

No further response has been received since the auto-acknowledgement.

• Matt

Comment 1

[Ryan Sleevi]

Robin (or other Sectigo staff): Could you please acknowledge this bug, to confirm an incident response is/will be provided? Given Sectigo’s historic past delays, I want to confirm this will be addressed.

Comment 2

[Robin Alden]

Hi Ryan,
We acknowledge this bug report and will provide an incident response.

We are gathering evidence to respond substantially, aiming to respond later today (Thursday 5th March), but no later than Friday 6th March.

Comment 3

[Robin Alden]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We received 3 problem reports to [e-mail] at
Wed 26-Feb 00:48 UTC
Wed 26-Feb 05:08 UTC
Wed 26-Feb 21:10 UTC

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Problem report #1 (our ref 00572387)

date&time stamp (UTC)	event
Wed 26-Feb 00:48	Problem report received for https://crt.sh/?id=2455920199
Wed 26-Feb 00:48	Sectigo CRM system created a case.
Wed 26-Feb 17:15 to 17:18	Sectigo support staff briefly reviewed case.
Wed 26-Feb 19:15	Sectigo support staff sent response: Thank you for bringing this to our attention, we will look into this right away.
Mon 02-Mar 19:41	Sectigo support staff escalated internally for assistance to help them verify if the private key for https://crt.sh/?id=2455920199 had been compromised.
Mon 02-Mar 19:44	Internal response indicates key is not proved to be compromised.
Fri 06-Mar 19:03	case is re-examined.
Fri 06-Mar 19:22	Sectigo support staff sent response: The private Key does not appear to be compromised.
Fri 06-Mar 19:24	Sectigo support staff sent response: disregard the prior email, looks like there was error when we checked before however we have now verified that the private key has indeed been compromised. The certificate will be revoked, thank you for bringing this to our attention.
Fri 06-Mar 19:52	Revoked.

Problem report #2 (our ref 00572465)

date&time stamp (UTC)	event
Wed 26-Feb 05:08	Problem report received for https://crt.sh/?id=2413850414
Wed 27-Feb 19:15	Sectigo support staff sent response: Thank you for bringing this to our attention, we will look into this right away.
Fri 06-Mar 18:47	case is re-examined.
Fri 06-Mar 19:08	Sectigo support staff sent response: Thank you for bringing this to our attention, the certificate will be revoked.
Fri 06-Mar 19:54	Revoked

Problem report #3 (our ref 00573105)

date&time stamp (UTC)	event
Wed 26-Feb 21:10	Problem report received for https://crt.sh/?id=683622319
Wed 26-Feb 21:10	Sectigo CRM system created a case.
Wed 26-Feb 23:37	Sectigo support staff change contact name on case to an unrelated email address and close the case.
Mon 02-mar 12:12	Sectigo CTO sent response: I did want to point out that - while we’re not great at publicising it - we do actually have a revocation ‘portal’ which would allow the possessor of a compromised key revoke the certificate ... https://secure.sectigo.com/products/RevocationPortal
Tue 03-mar 06:59	Reporter responds: This is interesting and potentially useful, however until it's listed in your CPS as an official channel for reporting certificate problems, there's no requirement that those reports are acted on, which makes it somewhat problematic to use. Also, the form isn't particularly amenable to automation
Fri 06-Mar 17:49	case is re-examined.
Fri 06-Mar 17:58	Sectigo support staff reopen ticket and change contact name back to the reporter's email address.
Fri 06-Mar 18:11	Sectigo support staff sent response: Thank you for bringing this to our attention. We will investigate immediately.
Fri 06-Mar 19:17	Sectigo support staff sent response: The private key does not seems to be compromised.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   This is not an issuance problem. We recognize the requirement to deal with problem reports in the timescales allotted and we have refocused our support staff on this task so that we achieve those timescales. We will take further steps as discussed below.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   The certificates themselves are not problematic, but the problem reports were concerning compromise of the private keys.
   The non-compliance being discussed here relates to our actions and responses to the problem reports rather than to the certificates themselves.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The certificates that were the subject of problem reports were:
https://crt.sh/?id=2455920199
https://crt.sh/?id=2413850414
https://crt.sh/?id=683622319

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Our initial investigation of the CRM tickets for these problem reports has not provided answers as to why the play-book was not followed for these reports.
We are discussing the issue with all of our support staff involved with the individual tickets, specifically not limited to those whose name(s) appeared in our emailed responses to the reporter. We have not yet had the opportunity to speak with everyone involved.
I will follow up on this ticket with further details in the coming days.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have reached out to our support staff to provide further guidance on how and why they must promptly deal with certificate problem reports in general and compromise reports in particular.
We have drafted a CPS update that includes details about our online revocation portal.
We will follow up on this ticket with details of further steps we take following the outcome of further investigations and discussions over the handling of these reports.
We will respond further within 1 week.

Comment 4

[mpalmer]

(In reply to Robin Alden from comment #3)

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have reached out to our support staff to provide further guidance on how and why they must promptly deal with certificate problem reports in general and compromise reports in particular.
We have drafted a CPS update that includes details about our online revocation portal.
We will follow up on this ticket with details of further steps we take following the outcome of further investigations and discussions over the handling of these reports.

What confidence do you have that the above steps will be sufficient to make it impractical for another problem report to be mishandled in the same way? I'd expect, for instance, perhaps something about QA review or reporting on problem reports which have not received an investigation report within a certain time period. Staff training is important, but humans are fallible creatures.

We will respond further within 1 week.

This sentence was posted 28 days ago...

Comment 5

[Ryan Sleevi]

I've updated Bug 1563579 regarding Sectigo's continued failure here.

Comment 6

[Robin Alden]

I apologize for the slow response.

Matt said

Staff training is important, but humans are fallible creatures.

We wholeheartedly agree. We have added a further automated revocation mechanism for compromised keys and we have documented in our CPS the mechanisms we provide that enable a speedy response to key compromise reports. Our recently updated CPS, available under https://sectigo.com/legal, includes the details of these, but for convenience I include a snippet from that document here.

• Sectigo also operates an automated revocation portal at https://secure.sectigo.com/products/RevocationPortal where Subscribers/Domain owners may revoke their certificates, or the public may report and revoke certificates for which the private key has been compromised.
• The public may also report and revoke certificates for which the private key has been compromised by using the ACME revokeCert method at this endpoint:
ACME Directory: https://acme.sectigo.com/v2/keyCompromise
revokeCert API: https://acme.sectigo.com/v2/keyCompromise/revokeCert

Comment 7

[mpalmer]

We have added a further automated revocation mechanism for compromised keys and we have documented in our CPS the mechanisms we provide that enable a speedy response to key compromise reports.

Automation is great, but for one-off key compromise reporting, I don't believe it is appropriate to require use of ACME. Which means that it is still important to maintain the quality and reliability of key compromise reports submitted via e-mail, so the question "What confidence do you have that the above steps will be sufficient to make it impractical for another problem report to be mishandled in the same way?" is still as relevant today as it was when I asked it two months ago.

It may also be worth highlighting, for Mozilla's consideration, another addition to the recently updated Sectigo CPS:

We are not able to respond in a timely manner to other types of report of key compromise.

Comment 8

[Ryan Sleevi]

(In reply to mpalmer from comment #7)

It may also be worth highlighting, for Mozilla's consideration, another addition to the recently updated Sectigo CPS:

We are not able to respond in a timely manner to other types of report of key compromise.

To be fair, this is within a section (1.5.2.1) for which they list a problem reporting email. As I read it, they are disclaiming timeliness for responses outside of those documented methods, which on its face doesn’t sound too unreasonable. If, for example, you were to ring up Sectigo after hours and leave a voice mail for a salesperson, I don’t think that would be fair to consider as in-scope. Of course, if they directed folks to call that salesperson, and then failed to deliver in a timely fashion, I would be deeply concerned.

That said, I do appreciate you flagging as this is the sort of thing we look for.

I do have to agree with Matt’s follow-up question though. To the extent manual methods are still permitted, and I do believe it’s both necessary and expected to both permit and potentially require them, how has that been addressed? I have extremely low confidence, given the failure to provide timely updates, that “more training” is the answer here.

Comment 9

[Robin Alden]

Matt,

Automation is great, but for one-off key compromise reporting, I don't believe it is appropriate to require use of ACME.

We agree.

Matt, Ryan,

Which means that it is still important to maintain the quality and reliability of key compromise reports submitted via e-mail

In general we disagree. That is why we have provided the automated webform-style interactive revocation tool at https://secure.sectigo.com/products/RevocationPortal.
This provides 2 simple tools for the submission of key compromise proofs that we can receive and process automatically and quickly.
If the key is not to be shared with us we require only that the reporter has the ability to use openssl from a command line using our suggested commands.
For those who cannot use openssl we will accept the private key pasted into a webform. While I acknowledge that receiving a compromised private key is not ideal it is hard to beat in terms of simplicity for the reporter.

We are not able to respond in a timely manner to other types of report of key compromise.

The point here is that, for key compromise reports, the role of our customer support staff should be to direct reporters towards our revocation portal and assist them to use it where they are otherwise unable to do so. That helps the problem reporter to submit their proof for prompt revocation which helps them and helps us. Our customer support staff will be far more effective in helping problem reporters to use the revocation portal which is simple and readily understood by our staff and, we believe, by even novice reporters who are genuinely in possession of a compromised key.

I do have to agree with Matt’s follow-up question though. To the extent manual methods are still permitted, and I do believe it’s both necessary and expected to both permit and potentially require them, how has that been addressed?

For key compromise reports (under BR 4.9.1.1 3) or invalidation of domain authorization or control (under BR 4.9.1.1 4) the only manual interaction we hope to have with reporters is to direct them to our automated webform revocation portal, or ACME end point as relevant, where they may effect the revocation they seek with the fastest possible response time. We are happy to have that interaction as many times as is necessary but any mechanism that depends on the sending and receiving of hand-written emails cannot operate at scale and will suffer errors.
For other types of problem report we still process them manually by email and we do not currently envisage having self-service portals for those other revocation reasons.

We would agree that training staff to type faster or more accurately cannot provide a solution to this problem. Automation, on the part of the CA at least, must be the only way that the expectation of reliable revocation at scale in a timely way can be met.
As the existence of these incident reports for non-timely or incomplete responses to revocation requests goes towards showing, us performing any form of process that requires manual steps to prove key compromise is an almost guaranteed route to failure given some volume of submissions.
Taking another aspect of this, the 24 hour timeline for revocation for key compromise set out in the BRs is achievable if the proof check is known and automated but is somewhere between hard and impossible to achieve on a 24x7x365 basis if each problem reporter can create a novel method, expressed in a different way each time to show proof of key compromise. Even where a particular reporter shows consistency, doing it the second time is almost as bad as the first. Where we get a new method we would create a process. When we automate that process it becomes part of our automated revocation service and only then can we hope to reliably meet the 24 hour timescale at volume. That automation is what we have done.

Comment 10

[mpalmer]

Unfortunately, I don't think you're going to be able to get to your platonic ideal, because there are more evidences of key compromise than are dreamt of in your philosophy. For my use-case, for example, if a key that is already known to be compromised by pwnedkeys is later used in a certificate, I cannot (easily) lay my hands on the private key material -- I keep all the existing private keys offline and encrypted, for what I hope are very, very obvious reasons. The ACME revocation protocol requires real-time access to the private key (to sign the request), and by the look of it, so does your revocation portal. So, I can't use either of those methods in a timely fashion to notify Sectigo of a compromise of a pre-pwned key that you got convinced to sign. A CA that required either millions of private keys to be available online at all times "just in case", or one that required key compromise revocation to be significantly delayed, would not, in my view, be making the web PKI ecosystem a better place.

Comment 11

[Ben Wilson]

Sectigo responded with an incident report in Comment 3 back in March 2020. Comment 9 explains Sectigo's approach to addressing the problem of receiving and processing revocation requests in a reasonable and timely manner, which I find sufficient and adequate. I have reviewed the current Sectigo CPS (v.5.2), section 1.5.2.1, and note that it explains various methods available for people to notify Sectigo of private key compromise. There are any number of other ways that one might want to use to establish that a private key has been compromised, but those methods need to be balanced based on a number of factors (e.g. automation, time involved, chance for human error, susceptibility to false positives, etc.) and a CA cannot be expected make everyone happy. Because I think that Sectigo has taken a reasonable approach, I am inclined to close this bug as fixed.

Comment 12

[Ben Wilson]

I am going to close this bug on or after 11-Aug-2020.