1650845 - Sectigo: Certificate Problem Report response issues

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Matthias]

Attachment: Timeline.txt

While communicating over the misissuance of https://crt.sh/?id=1777593900, Sectigo issued multiple incorrect or confusing statements.

The first two responses I received back from my initial problem report (one response at 2020-06-25 14:43, and one at 2020-06-26 19:53 UTC after double-checking with Sectigo) both assured that the cert "is revoked", even though the OCSP/CRL both would not list the certificate as revoked in the expected 1-hour/24-hour re-issue window as specified in their CPS. In my replies I explicitly mentioned that the OCSP still had the status "good", and the CRL did not yet contain the SN of the certificate. Only after explicitly noting that the certificate that was revoked seemed to be the replacement certificate (and not the original problematic certificate) I got a response detailing the revocation date of 2020-06-28; which turned out to be a revoke timestamp of 2020-06-29 02:37:40 UTC.

The certificate was eventually revoked with a timestamp of 2020-06-29 02:37:40 UTC, though "good" OCSP-responses were served at least until 2020-06-29 07:48:37 UTC. See attached file for a timeline of the communication with the timestamped communication.

This communication was sub-par of what I expect from a CA: I do not expect that a CA needs repeated reminders that their tatements are False, and I do not expect to need to ask explicitly for clarification about their False Statements as a result of the preliminary report (as per BR 2.9.5).

Comment 1

[Matthias]

Attachment: crt.sh screenshot with 'good' OCSP response at 2020-06-29 07:48:37 UTC

Comment 2

[Matthias]

Attachment: First manually found 'Revoked' response - note that the revocation date is before the previous check

Comment 3

[Nick France]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

25-Jun-2020 14:43 UTC - Email was received to SSL abuse address detailing problem of incorrect Subject information in certificate: https://crt.sh/?id=1777593900

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

25-Jun-2020 15:10 UTC - Certificate (https://crt.sh/?id=1777593900) revoked.

25-Jun-2020 15:12 UTC - Sectigo replied to the report from Mathias with the message as specified by Mathias:

that cert is waiting to be replaced with the correct information.
the current status of that cert is revoked.

"I try to verify this information, but neither CRL nor OCSP contains revocation information for this certificate in the 24 hours following the response."
The specific certificate reported by Mathias (https://crt.sh/?id=1777593900) was not revoked at the time. However, the response from our staff indicated it was.

The certificate was later revoked as part of a batch of revocations

Our system shows 3 certificates were issued as part of a single 'order' (in terms of how our system tracks them internally).

The original email from Mathias gave the crt.sh link which does not contain any Sectigo-specific identifiers.
Our staff would have used our internal order system to search. The default search date-range on the system begins at the start of the year (1/1/20).
Searching in the system for the FQDN 'alpha-data.com' on that range will pull up the two most-recent issuances of the certificate:
https://crt.sh/?id=2999092340
and
https://crt.sh/?id=1777593900

It would indicate that there is another, older issuance of the certificate even if that certificate was not immediately visible.

The first of the above two certificates (2999092340) was indeed revoked from the email Mathias sent.

29-Jun-2020 02:37 UTC - Certificate https://crt.sh/?id=1777593900 was revoked as part of batch of revocations related to bug 1645686.

We were concerned about the possibility that the 'good' OCSP response was served by our CDN for longer than we required.
We investigated with our CDN provider and found that for a short period, some requests for OCSP responses were served from out-of-date cached data.
We are working with the CDN provider to ensure that this is not repeated and also looking into additional or alternative providers for revocation services.

See also bug: 1645686 which will be shortly updated with more detailed information regarding the original problem of incorrect information in the Subject.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

NA

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   https://crt.sh/?id=2999092340
   https://crt.sh/?id=1777593900

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See 4.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The original misissuance was related to the issues in: 1645686
The miscommunication with the reporter was due to human error in searching for details of the reported certificate in our internal system, using a non-unique identifier (the FQDN) across a date-range

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Issues with validation are being addressed in: 1645686
Miscommunication: reminding staff who are handling abuse reports that they must confirm reports against our unique identifiers such as serial number, and not non-unique information such as Subject information or SAN entries.
Issues with outdated responses served from CDN: We believe this to have been a temporary and now-resolved issue with the CDN provider. However, other unrelated issues mean we are actively pursuing augmentation or replacement of our existing provider.

Comment 4

[Ryan Sleevi]

Miscommunication: reminding staff who are handling abuse reports that they must confirm reports against our unique identifiers

How is this different from what is explicitly called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report ?

For example, it’s not sufficient to say that “human error” of “lack of training” was a root cause for the incident, nor that “training has been improved” as a solution. While a lack of training may have contributed to the issue, it’s also possible that error-prone tools or practices were required, and making those tools less reliant on training is the correct solution. When training or a process is improved, the CA is expected to provide specific details about the original and corrected material, and specifically detail the changes that were made, and how they tie to the issue. Training alone should not be seen as a sufficient mitigation, and focus should be made on removing error-prone manual steps from the system entirely.

Comment 5

[Nick France]

We are still investigating what changes we can make to our customer-service systems for our staff responding to incident reports (that cannot be handled in an automated manner). This may require changes across our third-party ticketing system as well as our internal order-management systems.

We may combine the response with bug 1648717 and do a single (new) report with remediation for both.

Comment 6

[Ben Wilson]

When can we expect to see more information on this revamp?

Comment 7

[Nick France]

Ben: I've added an update on: https://bugzilla.mozilla.org/show_bug.cgi?id=1648717
Should we close one of these and continue any discussion on the other, or should I continue to maintain the same detail on both?

Comment 8

[Ben Wilson]

Please see Bug #1648717 for further discussion of this incident and its remediation.
I'll close this bug when the other bug is remediated, too.

Comment 9

[Rob Stradling]

Ben, are we still expected to continue to provide updates at least weekly in this particular bug until you close it? If so...

We have no further update.

Comment 10

[Ben Wilson]

No further updates are needed here. Thanks.

Comment 11

[Tim Callan]

(In reply to Ben Wilson from comment #8)

Please see Bug #1648717 for further discussion of this incident and its remediation.
I'll close this bug when the other bug is remediated, too.

Thank you for closing bug #1648717. I believe this bug is ready to close also.