1640310 - GoDaddy: Failure to revoke certificate with compromised key within 24 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Daniela Hood]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Mozilla.dev.security.policy on Thursday, May 21, 202020 3:10 PM.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Thursday, May 7, 2020 5:07 AM – We received a certificate problem report that indicated a possible key compromise.
Thursday, May 7, 2020 9:09 AM – 9:55 AM – The investigation started into details of the certificate listed to confirm the key was indeed compromised, contacted affected customer and contacted the problem reporter informing that the affected certificate was going to be revoked within 24 hours.
Friday, May 8, 2020 9:55 AM – Certificate was revoked due to a key compromise.
Tuesday, May 19, 2020 1:13 PM – We became aware of a thread on the Mozilla Dev Security Policy Group and began internal discussion to analyze potential issues.
Thursday, May 21, 2020 2:01 PM – Our internal analysis concluded the certificate was revoked within 24 hours of when we confirmed evidence of compromise, as was our practice. We posted our official answer to the thread and monitored closely for responses.
Thursday, May 21, 2020 3:10 PM – Another post was added, that indicated that our interpretation of policy was incorrect.
Thursday, May 21, 2020 3:30 PM – We regrouped and analyzed the applicable section of the BRs in light of the new information. Based on our revised understanding, we immediately updated our process as noted below.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We revised the process for certificate revocation to start the timer from the receipt of the certificate problem report instead of the conclusion of the investigation. Updated procedures have been communicated to personnel responsible for investigations and revocation actions.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

1 certificate that was identified with the issue.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=2366734355

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We interpreted the section of the revocation requirements related to reporting of a key compromise by third parties to require the certificate to be revoked within 24 hours of when the suspected key compromise was confirmed. We revised our understanding based on the Mozilla Dev Policy Form post, and understanding how the browsers interpret the requirement.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Our processes were immediately updated and communicated to personnel.

Comment 1

[Ryan Sleevi]

(In reply to Daniela Hood from comment #0)

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We interpreted the section of the revocation requirements related to reporting of a key compromise by third parties to require the certificate to be revoked within 24 hours of when the suspected key compromise was confirmed. We revised our understanding based on the Mozilla Dev Policy Form post, and understanding how the browsers interpret the requirement.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Our processes were immediately updated and communicated to personnel.

There seems to be a bigger issue here than is captured. That is, in Ballot SC6, explicit language was added that was meant to make it unambiguous the expectations. GoDaddy’s reply here still seems to be suggesting they’re ambiguous, and it’s a matter of how “browsers interpret the requirement”, where the requirement being discussed is:

The period from receipt of the Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time frame set forth in Section 4.9.1.1.

I’m not sure how that’s ambiguous or subject to interpretation. It raises questions in understanding how GoDaddy evaluates changes and reviews them, and calls into broader question other ballots that have introduced normative changes. It also raises questions in the sense of why the audit(s) didn’t catch this, which I think is a property of what controls GoDaddy had audited, since this does seem to be auditable both in a design phase and an execution phase.

I hope this doesn’t seem like me picking on GoDaddy, but I think to accept the explanation here, we need something more substantive in understanding what went wrong, given the plain language, and what can be done both by GoDaddy and other CAs to detect these sorts of “misinterpret plain language” issues.

For example, does GoDaddy believe the WebTrust detailed control reporting would have detected this? Would more normative guidance to auditors have helped this? Would different auditors have helped? Would an appendix of how CAs should implement this help? And if these would have helped, or wouldn’t have, what is GoDaddy doing about using or improving them?

Comment 2

[Daniela Hood]

GoDaddy acknowledges the inquiry and we will work to have a response to the community by Wednesday, June 3rd.

Comment 3

[Daniela Hood]

We appreciate the effort put into normative guidance and expressing requirements in terms of plain language. In reviewing our and other similar situations involving timing, we feel two things may contribute to misunderstandings. We also have a suggestion for how to potentially improve.

The first potential for misunderstanding is microfocus. When evaluating a specific issue, it may be necessary to reference more than one section of the requirements or even more than one document. In some cases, simply missing one cross reference can lead to a different outcome. The second area is the total document. With so many very specific timing requirements spread out across multiple documents, it is difficult to maintain a full picture on one's mind.

Our suggestion is a solution we are currently building, a timing matrix. The timing matrix brings together all the specific tasks that are on a timer, what time constraint exists, how it is measured, and cross reference to the section(s) where the requirement is referenced so we can quickly update it when there are changes. This is more approachable for the individuals that track compliance, but it is also a great communication medium to connect with teams performing manual tasks and engineers developing system functionality. The document could also be used as the basis for developing a review control, so we can ensure processes stay aligned and provide evidence for the audit. If this sounds like it would be useful to the community, we'd love to contribute our content.

Comment 4

[mpalmer]

That does sound like it would be very useful. I, for myself, look forward to seeing such a document published.

Comment 5

[Daniela Hood]

n add​ition and for transparency, we would like to report 13 more certificates that were revoked with the incorrect timeframe as pointed in this incident: https://bugzilla.mozilla.org/show_bug.cgi?id=1639798

All of those cases were related to the issue previously reported on this incident, and have already been addressed.

Comment 6

[Ben Wilson]

Hi Daniela,
Could you post a list of all the certificates covered by this incident report / disclosure? I don't see them listed elsewhere.
Thanks,
Ben

Comment 7

[Daniela Hood]

Here is the list of 13 certificates that were provided in Bug #1639798:

https://crt.sh/?id=2709508479
https://crt.sh/?id=2187681228
https://crt.sh/?id=907487329
https://crt.sh/?id=2170334409
https://crt.sh/?id=1516808581
https://crt.sh/?id=751943120
https://crt.sh/?id=1622196177
https://crt.sh/?id=2433522072
https://crt.sh/?id=1619018334
https://crt.sh/?id=2699348000
https://crt.sh/?id=1979163445
https://crt.sh/?id=1176645207
https://crt.sh/?id=1178545520​

After investigation on these certificates we identified that the root cause of the issue with these certificates is the one stated above. We are very committed to eliminate all possibilities of issues like this one from rising again. As mentioned before, we have created a timing matrix that has already proven to us internally to be a great tool when we need to reference the baseline requirements. GoDaddy's goal is to always improve and to contribute to the industry in maintaining a health ecosystem.

Comment 8

[mpalmer]

Is that timing matrix going to published for wider review and benefit?

Comment 9

[Daniela Hood]

Hello Matt,

We are currently working on the timing matrix for all the requirements, and we will be publishing them by August 31st​.

Comment 11

[Daniela Hood]

Hello,
Due to unforeseen circumstances we will postpone sharing the timing matrix to 09/15/20 in order to assure proper quality review.
We appreciate your understanding.

Comment 12

[Daniela Hood]

Attachment: Baseline Requirements - Timing Matrix.pdf

Hello,
Attached is the timing matrix created based on the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, Version 1.7.1, August 20, 2020.

Comment 13

[paul.leo.steinberg]

(In reply to Daniela Hood from comment #12)

Attached is the timing matrix created based on the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, Version 1.7.1, August 20, 2020.

Wow, even re bug 1662807, you still have not read SC31, which changed "Certificate Validity Period" to no longer start at "From issuance date": https://github.com/cabforum/documents/blame/6b870f92d4788a52c2bbc9d96a1db17751e906b1/docs/BR.md#L424 .

For a document with "assure proper quality review", this is pretty amazing (and it was so easy to spot!)! I slowly wonder if you have done any analysis on bug 1662807 at all.

Comment 14

[Daniela Hood]

Attachment: Baseline Requirements - Timing Matrix.pdf

Hello Paul,
We appreciate your comment and take it in the collaborative spirit in which our post was intended. While we view "from issuance date" to be substantially equivalent to notBefore, we do understand that other CAs may use this as a definitive field. Therefore, we have updated the document to match the exact verbiage of the Validity Period definition.

Comment 15

[Ben Wilson]

Thanks for providing the BR timing matrix. I believe this bug can be closed and intend to close it on or about 9-October-2020 unless there are additional issues or questions to address.