1649277 - DigiCert: Failure to provide a preliminary report within 24 hours.

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[George [:fozzie]]

I sent a problem report to [e-mail] and have yet to receive a preliminary report:

Saturday 27th June 12:55 UTC - I sent a report concerning:
https://crt.s/?q=CE6537ECE3373F038C89CAC35D0673C959BA2E43E6E069BEBC08F60234C65D7D https://crt.sh/?q=09084693897470D0517D1BF31CD0D99BC233E8C81F06F5870E5C6C2024664F5D

Due to having "CHANNEL ISLANDS" and "Channel Island" in the stateOrProvinceName field.

Saturday 27th June 15:03 UTC - I received a response saying that the investigation has started.

As of Monday 29th June 21:51 UTC I haven't had any further responses from DigiCert on this report.

Comment 1

[Brenda Bernal]

Incident Report – Mozilla Policy Violation

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

6/29/2020 – This case on Bugzilla was opened.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

6/27/2020 6:55 am MT/12:55pm UTC: Email Received from Reporter (George)
6/27/2020 9:03 am MT / 3:03 pm UTC: Acknowledged Reporter that we would investigate
6/29/2020 6:52 am MT/12:52pm UTC: Sent notification that cert will be revoked (Subscriber 1)
6/29/2020 6:56 am MT/12:56pm UTC: Sent notification that cert will be revoked (Subscriber 2)
6/29/2020 9:06 am MT/3:06pm UTC: Customer Revoked Certificate 1
6/29/2020 7:26 pm MT / 1:26 am UTC (next day): Sent update to Reporter that we would be revoking
7/02/2020 1:03 am MT / 7:03 am UTC: Certificate 2 Revoked by DigiCert

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We did miss providing a preliminary report to the reporter (George) within the 24 hour period. We are remediating the process to ensure this does not re-occur. Our plan to prevent this issue from happening is described in section 7.
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

First certificate issued: 01/30/2019
Last certificate issued: 05/05/2019

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1166013895
https://crt.sh/?id=1509393862

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This is a manual process and the Support person working the case sent the acknowledgement but missed the report back. This is due to human error.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We kicked off training to the entire Support team who work the problem cases about the requirement to also send preliminary report back to reporter within 24 hours and will include it as a checklist item on inbound cases, and explore any other automation possibilities.

Comment 2

[Ryan Sleevi]

Brenda: This incident report doesn't really analyze why the report was missed. It sounds like, by reading between the lines and what DigiCert didn't state directly, that there was no checklist to ensure a response was sent.

However, we know from time and time again that "more training" is not a systemic fix, it's continuing the same problematic practice, just more of it. I'm hoping a more thoughtful analysis about the underlying challenges can be applied, especially since DigiCert has had trouble with timely responses in other situations (e.g. Bug 1649880, Bug 1624504, Bug 1590171, Bug 1639801). What's being done systemically to address the challenges here?

Comment 3

[Brenda Bernal]

Hi Ryan, our plans to automate certificate problem reporting are shown here via the diagram: https://bugzilla.mozilla.org/show_bug.cgi?id=1639801#c14 and described here: https://bugzilla.mozilla.org/show_bug.cgi?id=1639801#c15. Although training is still being considered, we are augmenting the remediation with the automation described to reduce risk from human error which was the key factor in missing the problem reporting timeline.

Comment 4

[Ryan Sleevi]

Thanks Brenda. That's very useful.

I'm kicking this over to Ben. While I'm not seeing a concrete timeline in Bug 1639801, I do think the described flows seem like they would substantively address this issue.

Comment 5

[Brenda Bernal]

Thanks Ryan.

Jeremy indicated in this comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1639801#c17 the projected timeline to complete the automated revocation is beginning of October. This would be the same solution that would be applied for this issue.

Ben, let me know if you have any further questions before closing the bug.

Comment 6

[Brenda Bernal]

Update: We are expecting to post more information about the automated revocation tool that we are set to launch with more information early next week. This will include the notification for problem reporting.

Comment 7

[Brenda Bernal]

We provided an update on this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1639801. The tool went live. Is there anything else required before closing this bug?

Comment 8

[Ben Wilson]

I believe we can close it. Unless there are further issues to be raised/discussed, I'll schedule it for closure on 23-October-2020.

Comment 9

[George [:fozzie]]

Just a quick question Jeremy. In your diagram in bug 1639801 it says that revocation is assumed if a response isn't provided by the agent within 24 hours. I presume the actual time is less than 24 hours to avoid a compliance issue with sending a preliminary report within 24 hours? That is, if the timing is exactly 24 hours then the preliminary report would be sent out after the 24 hour deadline.

Comment 10

[Jeremy Rowley]

Correct - it's actually set at 22 hours from when submitted. .

Comment 11

[George [:fozzie]]

Alright thanks. I saw you mentioned the 22 hours for the compromised keys, wanted to make sure it also applied to preliminary reports. I don't believe I have any more questions.

Comment 12

[Ben Wilson]

I'll close this next Wednesday, 28-Oct-2020 unless there are other issues to raise and discuss.