1942877 - GoDaddy: Delayed revocation

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Hanno Boeck]

I would like to report a delayed revocation by GoDaddy.

I have reported a number of certificates with compromised keys due to the Fortinet leak to GoDaddy on Friday and Saturday. They are still not revoked.

I first tried reporting 31 compromised certs on Friday morning to GoDaddy's problem reporting mail address. The mail address does not accept mails with attachments, which I already reported as https://bugzilla.mozilla.org/show_bug.cgi?id=1942241

In response to that bug, I was contacted on Friday afternoon by someone from GoDaddy asking to send the keys. I immediately replied to the mail with the certs+keys attached. On Saturday, I sent a report about three more affected certs to the same address.

As of now, the certs are still not revoked.

Affected serials reported in first batch: 08E258BAE00DC858 0973C18232DBBA 164FFF2684C46EFE 206D1410515237A6 25A319CDFCBED56E 2C3D628E5395BC64 4B8CA75FCD47C8E8 580EAB40DF2F3465 5DF0B0794545B7A4 64CD8A867999F66D 6AF26A9F76A6CFBB 7148A89B310FD3F4 80941F7568A05FA6 81FD3CE71E6C0580 82A6B78CD49FB7C3 872F88DEA07C7E64 877510D64C638403 8F12821D52981FE7 98A07CF580400985 A1A7DF22B21085AD A240CEBED2255690 A435F2F503D306F7 C18A824FACEF835C C80616C1E851EE6B C8FE0E5E55A522A1 CA116061702B6B3A D01AA66B45ABFC84 D30EDAEF2C811BBC D46290556CFB92DC D5F21DEAC420D0E7 DF830533202772BB E7C7E3141BE98C2A
Reported in second batch: 551B592F76402BB1 A2FA0EB1703A4A9B B76173DC511EC281

Comment 1

[Steven Deitte]

Apologies for missing these emails, they were unfortunately quarantined by our email servers so we were unaware anything was sent. We have worked with our email provider to fetch the quarantined emails and are working through revoking affected certificates. We will draft a full report with action items to improve this process in the coming days.

Comment 2

[Brittany Randall]

Incident Report

Summary

GoDaddy provides an email address ([e-mail]) for reporting any issues with certificates including compromised keys. Malware and anti-virus scanning enabled on [e-mail] email address inadvertently blocked the ability to send emails of certain file types containing compromised keys.

Section 4.9.3 of the Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates states “The CA SHALL maintain a continuous 24x7 ability to accept and respond to revocation requests and Certificate Problem Reports.” While [e-mail] email address was available and our teams were staffed to review and respond timely, malware filters inadvertently prevented us from receiving information that was relevant to the problem report.
GoDaddy has worked with the problem reporter to ensure we received all problematic keys and have revoked certificates accordingly.

Impact

32 certificates were delayed being revoked for 4 days from first attempt of reporting.
3 certificates were delayed being revoked for 3 days from first attempt of reporting
See Appendix for details of affected certificates. Note: 4 certificates were already in an expired status at the time we took action to revoke

Timeline

All times are UTC.

2025-01-17
-07:00: Bugzilla 1942241 filed advising CPR intake email address ([e-mail]) was not accepting attachments
-14:26: GoDaddy PKI Eng Manager sent direct email from direct employee account asking reporter to send the emails directly in thread as a reply.
-18:12: PKI Policy reached out to central email team to confirm receipt of blocked email from reporter due to malware filters
-19:06: Central email team confirmed there was an email blocked from the reporter and identified a security setting (malware filter) that was blocking certain filetypes from being received
-20:01: Updated security settings for the intake email alias of [e-mail] to allow for blocked filetypes both as direct attachments and within compressed file attachments
2025-01-21
-17:33: Bugzilla 1942877 filed advising delayed revocation for 35 Certificates with compromised keys
-17:55: PKI Policy reached out to central email team to check for blocked emails on direct employee accounts
-18:21: Central email team confirmed two quarantined emails sent to direct employee email aliases that held the compromised key data. Started retrieval process.
-19:55: PKI Policy received contents of the two quarantined emails from email team and forwarded them to PKI Engineering team to confirm compromised keys and scrub our database for impacted certificates issued based on these keys.
-23:51: All affected certs surfaced and revoked by PKI Engineering team
2025-01-22
-06:58 Emailed reporter confirming all impacted certificates have been revoked

Root Cause Analysis

GoDaddy’s configuration to block or quarantine emails was actively scanning for malware or viruses on the email address used for reporting [e-mail] which meant that emails with specific file types within the attachments, in this case specifically .key and .crt files were bounced back to senders.

Separately, when attempting to work directly with the reporter who identified this issue, it was found that our individual employee email accounts at GoDaddy quarantine emails that had findings with the same malware and virus scanning rather than sending a bounce-back message to the sender. This further delayed our ability to process the compromised keys.

Lessons Learned

Email is not an ideal solution for certificate problem reporting given the nature of attachments that are in some cases required to be sent. GoDaddy will be introducing an HTML form for customers to report certificate problems.

What went well

• GoDaddy was able to retrieve the quarantined emails sent directly to employees’ email addresses and swiftly process the revocations after verifying the compromised keys
• GoDaddy was successful in updating the malware configuration settings of the [e-mail] email address and subsequently received new problem reports with similar file attachments that were initially blocked by the same reporter.

What didn't go well

• Attempts to work directly with the problem reporter after the initial Bugzilla report were made were made difficult by similar malware configuration with a different outcome (bounce-back message to the user compared to quarantined email). Had we anticipated the quarantined messages we could have processed these reported compromised keys faster.

Action Items

Action Item	Kind	Due Date
Update email malware configuration for [e-mail] to allow additional file types (.key and .crt) to be received	Prevent	2025-01-17
Review [e-mail] traffic history as far back as email client can access to confirm this was an isolated incident	Mitigate	2025-02-28
Add monitoring to the [e-mail] email address for alerts on any blocked or quarantined emails enabling us to identify if there are other problem use cases not identified in this issue	Detect	2025-02-28
Introduce a webpage form solution for certificate problem reporting to the GoDaddy CA	Prevent	2025-03-31

Appendix

Details of affected certificates

1. https://crt.sh?id=16011662993
2. https://crt.sh?id=15961831627
3. https://crt.sh?id=15873467431
4. https://crt.sh?id=15828801979
5. https://crt.sh?id=15822080599
6. https://crt.sh?id=15783775181
7. https://crt.sh?id=15769792387
8. https://crt.sh?id=15596941459
9. https://crt.sh?id=15366101829
10. https://crt.sh?id=15261492836
11. https://crt.sh?id=14990298071
12. https://crt.sh?id=14847657914
13. https://crt.sh?id=14096761501
14. https://crt.sh?id=14050495313
15. https://crt.sh?id=14007375139
16. https://crt.sh?id=13911555688
17. https://crt.sh?id=13877459492
18. https://crt.sh?id=13825929251
19. https://crt.sh?id=13796514692
20. https://crt.sh?id=13594686424
21. https://crt.sh?id=13367412751
22. https://crt.sh?id=13345035439
23. https://crt.sh?id=12268120520
24. https://crt.sh?id=12196951952
25. https://crt.sh?id=11927203792
26. https://crt.sh?id=11856747991
27. https://crt.sh?id=11843042538
28. https://crt.sh?id=11637804319
29. https://crt.sh?id=16137109941
30. https://crt.sh?id=11700794405
31. https://crt.sh?id=14035675461
    The following four certificates were expired at the time when we went to revoke.
32. https://crt.sh?id=11409404252
33. https://crt.sh?id=11406293742
34. https://crt.sh?id=11408322066
35. https://crt.sh?id=11407756350

Comment 3

[Steven Deitte]

Update on action items -

Action Item	Kind	Due Date	Status
Update email malware configuration for [e-mail] to allow additional file types (.key and .crt) to be received	Prevent	2025-01-17	Complete
Review [e-mail] traffic history as far back as email client can access to confirm this was an isolated incident	Mitigate	2025-01-29	Complete
Add monitoring to the [e-mail] email address for alerts on any blocked or quarantined emails enabling us to identify if there are other problem use cases not identified in this issue	Detect	2025-02-13	Complete
Introduce a webpage form solution for certificate problem reporting to the GoDaddy CA	Prevent	2025-03-31	Ongoing

Comment 4

[Steven Deitte]

Update on our action items - we have made good progress on the webpage form solution for certificate problem reporting. The coordination across multiple engineering teams for the automation of ticket creation for our RA processes is taking a bit longer than anticipated. We've updated the due date to April 30th, 2025 which we are confident we can hit.

We have continued to closely monitor our practices@ email address for any issues with problem reporting to ensure we maintain availability until we can get the improved webpage rolled out.

Action Item	Kind	Due Date	Status
Update email malware configuration for [e-mail] to allow additional file types (.key and .crt) to be received	Prevent	2025-01-17	Complete
Review [e-mail] traffic history as far back as email client can access to confirm this was an isolated incident	Mitigate	2025-01-29	Complete
Add monitoring to the [e-mail] email address for alerts on any blocked or quarantined emails enabling us to identify if there are other problem use cases not identified in this issue	Detect	2025-02-13	Complete
Introduce a webpage form solution for certificate problem reporting to the GoDaddy CA	Prevent	2025-04-30	Ongoing

Comment 5

[Steven Deitte]

Update on our action items - we have rolled out our web based problem reporting page which can be found at https://sec.godaddy.com/report-certificate. I will follow up with a closure summary for this incident shortly.

Action Items

Action Item	Kind	Due Date	Status
Update email malware configuration for [e-mail] to allow additional file types (.key and .crt) to be received	Prevent	2025-01-17	Complete
Review [e-mail] traffic history as far back as email client can access to confirm this was an isolated incident	Mitigate	2025-01-29	Complete
Add monitoring to the [e-mail] email address for alerts on any blocked or quarantined emails enabling us to identify if there are other problem use cases not identified in this issue	Detect	2025-02-13	Complete
Introduce a webpage form solution for certificate problem reporting to the GoDaddy CA	Prevent	2025-04-30	Complete

Comment 6

[Steven Deitte]

Report Closure Summary

• Incident description: Revocation for key compromise was delayed due to complications with the problem reporting email address used by GoDaddy
• Incident Root Cause(s): Email malware scanning blocked problem reports reaching the GoDaddy support inbox
• Remediation description:
  -- Updated email configuration for certificate problem reporting to allow more file types as attachments
  -- Improved monitoring on the email address used for certificate problem reporting
  -- Introduced a web based form for certificate problem reporting
• Commitment summary: GoDaddy will continue to monitor our email address and web based form to ensure certificate problems reported are promptly processed.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 7

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, this bug will be closed on approximately 2025-05-08.