1742602 - GoDaddy: Reported TLS Certificate Private Key Exposure

Status: RESOLVED DUPLICATE of bug 1742657

(CA Program :: CA Certificate Compliance, task)

Description

[Ryan Dickson]

Steps to reproduce:

An SEC filing made on November 22, 2021 indicates a private key exposure for a subset of GoDaddy customers.

Link: https://www.sec.gov/Archives/edgar/data/1609711/000160971121000122/gddyblogpostnov222021.htm

A notification was also made on MDSP at https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/bynogtzVDCQ/m/9wHdhCBCBwAJ

Actual results:

N/A

Expected results:

N/A

Comment 1

[Matthias]

Ryan, could you specify if / why you think this is an incident? Not that I have an opinion either way (it could or could not be an incident) or wouldn't appreciate a more in-depth explanation, but the only "problem" that is reported ('private key exposure of subscriber certificates') is not forbidden by the BR nor the MRSP; there are well-known procedures for handling this available in the BR.

I can only think of three reasons why this could be an incident:

1. The revocation of affected certificates was not within 24 hours of the CA becoming aware (aka receiving the message / discovering the issue / ..., whichever first) (BR s4.9.1.1);
2. The CRLs were not updated in time with the revocation data (BR s4.9.7, tangentally related to 2).
3. The CA generated keys for impacted subscriber certificates (forbidden as per BR s6.1.1.3);

3 is unrelated to the problem in the SEC filing and somewhat difficult to prove, while 2 depends on the 7-day re-issuance of CRLs (the issue only came to light yesterday, so highly unlikely that that is currently a problem).

For 1, I can see that they mention a discovery of unauthorized access dated on the 17th of November, but that does not mean that they found out the certificates were compromised that same day.

Anecdotal evidence: https://crt.sh/?id=5646500305&opt=ocsp was revoked within 24 hours of the SEC publication (at 2021-11-22 09:19:11 ET / 14:19:11 UTC, versus a revocation date in OCSP response of 2021-11-23 09:40:07 UTC, a time difference of 19h20m). This allows for a little less than a 5-hour detection-to-issue-publication window without violating BR s4.9.1.1

Comment 2

[Ryan Dickson]

Hi Matthias,

My immediate hope in creating this Bug was for GoDaddy to share additional information to help us determine whether an incident occurred or not. Given that an unauthorized third party had access to a system containing end-entity private keys, it seemed reasonable that we could at least suspect an incident (key compromise) took place. Depending on the level of auditing and logging enabled on the targeted system, it may also be impossible to prove that key compromise did not occur (and even then, it raises possible questions about the integrity of the audit/log data).

As noted in the Chrome Root CA Policy:

Chrome requires that any suspected or actual compliance incident be promptly reported and publicly disclosed upon discovery by the CA, along with a timeline for an analysis of root causes, regardless of whether the non-compliance appears to the CA to have a serious or immediate security impact on end users.

As you indicated, confirming the suspected incident would result in a violation of the Baseline Requirements Section 4.9.1.1 ("Reasons for Revoking a Subscriber Certificate") due to either of the following requirements:

1. The CA obtains evidence that the Subscriber's Private Key corresponding to the Public Key in the Certificate suffered a Key Compromise (revocation required within 24 hours); or
2. The CA is made aware of a demonstrated or proven method that exposes the Subscriber's Private Key to compromise or if there is clear evidence that the specific method used to generate the Private Key was flawed. (revocation required within 5 days)

Given 1742657, GoDaddy has evaluated the latter as being most appropriate. What's unclear to me, at this point, is whether or not #1 is more appropriate.

Thanks,
Ryan

Comment 3

[Brittany Randall]

Apologies for not seeing this bug before posting 1742657 (linked above). We will be providing a full incident report on 1742657 in the coming week. Ryan D., are you okay if we close this bug as a duplicate? We will of course be considering community feedback as we prepare our full incident report in the other bug.

Comment 4

[Ryan Dickson]

No issues with that, Brittany! Thank you for checking.

• Ryan

Comment 5

[Matthias]

(In reply to Ryan Dickson from comment #2)

Hi Matthias,

Hi Ryan,

Regarding GoDaddy's incident itself, I agree that bug 1742657 is the right place to discuss the issue(s) at hand; I hadn't yet seen that issue pop up when I posted comment 2.

The rest of this comment is on the topic of the Chrome Root Store's expectations from CAs, as I understand (or don't understand) the Chrome Root CA Policy (CRCAP), based on your previous comments. Note that my understanding is mostly based on my assumption that your comments are written with the Chrome Root Store hat on, and I don't know if this is the right forum for that, so please correct me if I'm wrong.

---

Given that an unauthorized third party had access to a system containing end-entity private keys, it seemed reasonable that we could at least suspect an incident (key compromise) took place.

I understand that key compromise is a potential incident, but to the best of my knowledge, subscriber certificate key compromise has never been considered a CA incident; key compromise is only an incident when a CA's signing keys are compromised.

Do you consider the key compromise to be an incident because GoDaddy, as a CA, was the manager of the keys and should have protected the keys better? If that is not why this is an incident, then do you expect each CA to create incidents because their subscribers mismanaged their private keys and filed for key compromise at the CA's problem report contact?

As noted in the Chrome Root CA Policy:

Chrome requires that any suspected or actual compliance incident be promptly reported and publicly disclosed upon discovery by the CA, along with a timeline for an analysis of root causes, regardless of whether the non-compliance appears to the CA to have a serious or immediate security impact on end users.

Where CRCAP qualifies what an incident is:

Failure of a CA to meet their commitments, as outlined in their CP/CPS and the Baseline Requirements, is considered an incident, as is any other situation that may impact the CA’s integrity, trustworthiness, or compatibility.

Does this 'integrity, trustworthyness, or compatibility'-clause extend to non-CA business of a CA? E.g. hypothetically, if AMTrack decides to start a public CA, and then one of their trains derails (arguably impacts trustworthyness / integrity of AMTrack's transport department, thus AMTrack); should they file for an incident according to the CRCAP? Or a hypothetical DigitalOcean CA, of which subscriber certificates issued to the public are found to be compromised due to public access policies in their blob store cloud solution?

So my questions are basically: Where do you consider the line between a "CA incident" where an incident report is required, and a "non-CA incident" where no report is required, with special attention for when the CA is part of a larger organisation?
If a leak at one branch of a company results in mass revocation of that branch's leaf certificates (that were issued by the CA operated in another branch of the same company), is that in itself a CA incident?

Thanks,
Matthias

Comment 6

[Ryan Dickson]

Hi Matthias,

I appreciate the continued discussion. Please consider my posts related to this incident in an official capacity, and I’ll be more clear moving forward as I continue my involvement with the community.

Some added perspective, below:

Do you consider the key compromise to be an incident because GoDaddy, as a CA, was the manager of the keys and should have protected the keys better?

No. However, in this case, my concern was elevated because I felt there was a clear incident (violation of BRs Section 4.9.1.1), and until it was also clear the attack on the provisioning system did not adversely impact the integrity, trustworthiness, and/or compatibility of other PKI systems, it felt safest for Chrome’s user community to suspect otherwise.

Thought process:

• When I created this bug, we lacked sufficient detail related to the scope of the incident described in the SEC filing.
• The filing indicated a compromised password allowed an unauthorized third party to access a GoDaddy system responsible for storing end-entity private keys. It seemed safest to file a bug to obtain more detail on whether this compromise might have also extended to the CA, e.g., by a shared network security perimeter around the provisioning system and the CA, or by credential reuse across the two systems.
• GoDaddy later added some context related to the security of the CA in bug 1742657, but that wasn't available when I created this bug.

To be direct:

If that is not why this is an incident, then do you expect each CA to create incidents because their subscribers mismanaged their private keys and filed for key compromise at the CA's problem report contact?

No. This is an incident because GoDaddy failed to revoke the certificates whose corresponding private keys were compromised by the breach in a timely manner - as confirmed by the creation of 1742657.

Again, I had additional interest in this scenario due to a combination of the nature of the compromise, the targeted system's perceived relationship with an issuing CA, and its assumed location within GoDaddy's network security perimeter.

With regard to:

Does this 'integrity, trustworthyness, or compatibility'-clause extend to non-CA business of a CA? E.g. hypothetically, if AMTrack decides to start a public CA, and then one of their trains derails (arguably impacts trustworthyness / integrity of AMTrack's transport department, thus AMTrack); should they file for an incident according to the CRCAP? Or a hypothetical DigitalOcean CA, of which subscriber certificates issued to the public are found to be compromised due to public access policies in their blob store cloud solution?

and

Where do you consider the line between a "CA incident" where an incident report is required, and a "non-CA incident" where no report is required, with special attention for when the CA is part of a larger organisation? If a leak at one branch of a company results in mass revocation of that branch's leaf certificates (that were issued by the CA operated in another branch of the same company), is that in itself a CA incident?

I understand and agree with your underlying point. CAs need clear, consistent, and objective guidance related to the expectations of root programs to distinguish events that require incident reports and those that don't. As we continue operationalizing the Chrome Root Program, we're planning updates to our existing policy. As we do, we'll be mindful of your concerns and attempt to distinguish reporting requirements for suspected and confirmed incidents more clearly. In this instance, my intent in filing the bug was to determine whether an incident occurred (which seemed likely based on 4.9.1.1 at least) and, if so, what type, rather than to indicate that our policy considers the EE key compromises an incident that required reporting.

Would you please let me know if you have any other questions?

Thanks,
Ryan

Comment 7

[Matthias]

(In reply to Ryan Dickson from comment #6)

Hi Matthias,

I appreciate the continued discussion. Please consider my posts related to this incident in an official capacity, and I’ll be more clear moving forward as I continue my involvement with the community.

Some added perspective, below:

Do you consider the key compromise to be an incident because GoDaddy, as a CA, was the manager of the keys and should have protected the keys better?

No. However, in this case, my concern was elevated because I felt there was a clear incident (violation of BRs Section 4.9.1.1), and until it was also clear the attack on the provisioning system did not adversely impact the integrity, trustworthiness, and/or compatibility of other PKI systems, it felt safest for Chrome’s user community to suspect otherwise.

Ah, ok, then I misunderstood what you meant with "[...] it seemed reasonable that we could at least suspect an incident (key compromise) took place"; thanks for clarifying.

Would you please let me know if you have any other questions?

No further questions.

Thanks,
Matthias

Comment 8

[Brittany Randall]

Marking this bug as a duplicate. Please refer to bug 1742657 for tracking.