1942879 - Globalsign: Delayed revocation

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Hanno Boeck]

I would like to report a delayed revocation incident for the CA Globalsign.

I reported 6 certificates with compromised keys due to the Fortigate leak on Friday to Globalsign's problem reporting address, with the certs and private keys attached.
I reported 2 more affected certificates on Saturday.

Globalsign has not reacted to these reports.
All certificates are still unrevoked at the time of this report.

Serials report 1: 2D1FF0A25269872917AE2A3E 3673DF60BD627D8D239F0C8F 47EF73CFD767281BC28576A6 4B4D0EC15C937219DEEA19A1 5D1B50D28EE11A77CFA3EED5 5E54F251877E3E58A85B46E9
Serials report 2: 13EF4598A344494FC86C9167 1F95543B714211C63F1E5BBA

Comment 1

[Christophe Bonjean]

Thank you for reporting this, investigation has started and we have reached out to the reporter.

We will provide a detailed incident report as soon as we have concluded our analysis, but no later than Tuesday, 28th of January.

Comment 2

[Hanno Boeck]

I have received an email from globalsign asking me to re-send the keys "as an attachment in a commonly used attachment type (i.e. tar, tar.gz, zip)".

I have immediately replied with a zip with all the affected certs+keys. (My original report included them as .tar.xz.)

It appears that this is a generic problem, as mails to another CA (godaddy) also have not reached them due to attachments containing cryptographic keys. See also:
https://bugzilla.mozilla.org/show_bug.cgi?id=1942877
https://bugzilla.mozilla.org/show_bug.cgi?id=1942241

In both cases, it appears the CAs are using emails hosted with outlook mail protection. Notably, a mail with a zip attachment also has not reached godaddy.

It appears that this may be a larger problem, as I assume many CAs likely use such a setup. If those do not deliver attachments including certs or private keys, this likely affects more CAs.

Comment 3

[Wayne]

(In reply to Christophe Bonjean from comment #1)

Thank you for reporting this, investigation has started and we have reached out to the reporter.

We will provide a detailed incident report as soon as we have concluded our analysis, but no later than Tuesday, 28th of January.

Globalsign still need to provide a preliminary report within 72 hours of notification.

(In reply to Hanno Boeck from comment #2)

In both cases, it appears the CAs are using emails hosted with outlook mail protection. Notably, a mail with a zip attachment also has not reached godaddy.

It appears that this may be a larger problem, as I assume many CAs likely use such a setup. If those do not deliver attachments including certs or private keys, this likely affects more CAs.

Notably this isn't the first time a mail filtering issue has affected CPRs:
https://bugzilla.mozilla.org/show_bug.cgi?id=1886626

As CAs are required to follow their peers' reports and identify if they were similarly affected I look forward to Globalsign's full report as due no later than Tuesday, 28th of January.

Comment 4

[Claves Nostrum]

Reading the above and doing a dig on GlobalSign's DNS records they indeed seem to be using o365. If am not mistaken .tar.xz is one of the file types that is typically blocked by e-mail providers. Hanno, surely you must have received a bounce or something that your e-mail with report could not have been delivered?

Comment 5

[Hanno Boeck]

I have not received any error mail or bounce in this case. (I did receive a bounce mail in the case in bug #1942241, but there, neither the zip nor the .tar.xz was accepted.)

Comment 6

[Christophe Bonjean]

The affected certificates of the two Certificate Problem Reports mentioned in this bug were revoked within 24 hours of receipt of the zip file which was received on 22/01/2025 at 10:51 UTC (as per Comment 2).

Link	Revocation (dd/mm/yyyy) - time UTC
https://crt.sh/?id=16264492889	23/01/2025 08:28
https://crt.sh/?id=14004256761	23/01/2025 08:29
https://crt.sh/?id=12778752647	23/01/2025 08:24
https://crt.sh/?id=12355066247	23/01/2025 08:27
https://crt.sh/?id=13583486868	22/01/2025 13:21
https://crt.sh/?id=12740137156	23/01/2025 08:26
https://crt.sh/?id=13081371387	23/01/2025 08:23
https://crt.sh/?id=14851594528	23/01/2025 08:25

In the meantime we received a third report by the same reporter, for which we were able to confirm key compromise. The revocation for the affected certificates is scheduled to be completed by 23/01/2025 16:15 UTC.

Comment 7

[Christophe Bonjean]

Incident Report

Summary

Two Certificate Problem Reports with private keys for 8 certificates were sent by the reporter but not successfully received by GlobalSign due to the attachment (.xz) being recognized as malware and blocked by filtering settings. The reporter did not receive a notification that the Certificate Problem Reports were not received, which caused a delay in revocation.

Impact

8 certificates with a compromised key were not revoked within 24 hours of the original Certificate Problem Reports from the reporter.

Timeline

All times are in UTC.

Date (dd/mm/yyyy)	Description
26/08/2024 07:32	Previous certificate problem report flagging key compromise from the same reporter received and processed successfully.
17/01/2025 07:45	Creation of Bugzilla 1942241 GoDaddy: Revocation process is unusable due to contact address not accepting attachments
17/01/2025 10:25	Creation of Bugzilla 1942270 SSL.com: Revocation process requires submission to a form that is unusable
17/01/2025 06:53	Email from reporter with subject “Request for Globalsign certificate revocation / FortiGate” failed to be delivered due to blocked attachment type. Reporter did not receive notification it was not delivered.
17/01/2025 10:48	Compliance team reviews 1942270 and 1942241, performs testing of emails with attachments and confirms that attachments can be received. Concludes GlobalSign is not affected by the issue.
17/01/2025 10:57	Compliance informs the relevant teams that there is a high potential of receiving key compromise reports.
18/01/2025 13:41	Email from reporter with subject “Request for certificate revocation (part2) / Globalsign / Fortigate” failed to be delivered due to blocked attachment type. Reporter did not receive notification it was not delivered.
21/01/2025 17:38	Compliance team is notified of a Bugzilla ticket having been created for GlobalSign.
21/01/2025 23:07	After risk assessment, removed the .xz attachment type from the anti-malware filter, allowing emails with attachments of this file type to be received.
22/01/2025 10:02	Key blocklist updated with potentially compromised keys for the certificates in this Bugzilla ticket.
22/01/2025 10:16	Compliance team reaches out to reporter for proof of compromise and starts notifying potentially affected Subscribers.
22/01/2025 10:52	Email received with zip file including relevant key compromise evidence.
22/01/2025 13:56	Compliance team confirms key compromises to reporter and starts replacement process with affected Subscribers.
22/01/2025 14:48	Established that the non-delivery report (NDR) was not successfully sent to the reporter.
22/01/2025 16:19	Identified configuration setting preventing NDR from being delivered.
22/01/2025 17:23	Updated configuration setting, confirmed successful delivery of NDR.
23/01/2025 08:29	All affected certificates from the Certificate Problem Reports revoked.

Root Cause Analysis

Upon being notified of the deliverability issues of other CAs, we first established we were not affected by Bugzilla 1942270 and further investigated for Bugzilla 1942241. In the latter, the reporter raised issues with deliverability of emails with attachments (zip, tar). We tested this with our reporting email address. Since we confirmed that keys and attachments of the previously reported types could be received, we concluded that we were not affected by the issue.

When this Bugzilla ticket was raised, we analyzed the deliverability of the Certificate Problem Reports and observed that the emails included an attachment with the extension .xz. The .xz extension is commonly recognized as malware and is included in the default attachment filter of our email provider's anti-malware policy. The emails were therefore not received by our report abuse team.

The system also generated a non-deliverability report (NDR), which stated that the sender was informed of the failure to deliver the email. However, upon further testing, we identified that the NDR report was not successfully delivered to the reporter due to a setting in our email configuration for this particular mailbox. As a result, the reporter was not informed about the failed email delivery for the Certificate Problem Reports.

Lessons Learned

What went well

• The internal escalation process ensured that the root cause was identified rapidly.

What didn't go well

• The non-delivery report (indicating to the reporter that the attachment was blocked) was not sent, which was a false indication to the reporter that the message was delivered.

• We tested and confirmed sending attachments to determine if we were affected by the same issue as the other Bugzilla tickets, but at the time did not review the non-delivered emails to the report-abuse mailbox.

Where we got lucky

• A limited number of certificates was impacted.

Action Items

Action Item	Kind	Completion
Removed .xz attachment type from the filter.	Prevent	21/01/2025 23:07
Updated email configuration for Non-delivery reports.	Prevent	22/01/2025 17:23
Reviewed the list of blocked file types to identify any other file type that might potentially be used to report / transmit a collection of compromised keys.	Prevent	22/01/2025 19:14
Reviewed all failed deliveries to the report-abuse mailbox within the past 90 days for missed Certificate Problem Reports.	Detect	23/01/2025 13:17

Appendix

Details of affected certificates

Link	Revocation (dd/mm/yyyy)
https://crt.sh/?id=16264492889	23/01/2025 08:28
https://crt.sh/?id=14004256761	23/01/2025 08:29
https://crt.sh/?id=12778752647	23/01/2025 08:24
https://crt.sh/?id=12355066247	23/01/2025 08:27
https://crt.sh/?id=13583486868	22/01/2025 13:21
https://crt.sh/?id=12740137156	23/01/2025 08:26
https://crt.sh/?id=13081371387	23/01/2025 08:23
https://crt.sh/?id=14851594528	23/01/2025 08:25

Comment 8

[Mike Shaver (:shaver emeritus)]

Christophe, thank you for the incident report. The detailed timeline is great.

One thing that I think is missing from the report, however, is mentioned by Wayne above:

(In reply to Wayne from comment #3)

Notably this isn't the first time a mail filtering issue has affected CPRs:
https://bugzilla.mozilla.org/show_bug.cgi?id=1886626

As CAs are required to follow their peers' reports and identify if they were similarly affected I look forward to Globalsign's full report as due no later than Tuesday, 28th of January.

Could you update the report to outline what Globalsign did in reviewing bug 18886626 regarding email deliverability of CPRs, and why that was insufficient to prevent this incident?

Also, the action items don’t seem to include any ongoing validation of the deliverability of CPRs via email. In order to prevent a change in email configuration from causing another incident of this nature, something along those lines feels appropriate to me. Do you agree?

Comment 9

[Christophe Bonjean]

The delivery issue in Bugzilla 1886626 was caused by the Certificate Problem Report being marked as junk. At the time of that bug, we reviewed the spam filtering settings for the report-abuse mailbox and confirmed that the spam filter was configured to be permissive and emails marked as junk were being delivered.

Based on this analysis, we concluded we were not affected by the issue in Bugzilla 1886626. Since non-delivery reports are not sent for emails marked as junk and the scope of the issue was limited to junk filtering, the deliverability of NDRs was not part of this analysis.

We will implement validation of deliverability of CPRs on a quarterly basis or on changes to the mailbox configuration, during which we will test accepted and filtered file types. This will be implemented by February 4, 2025.

Comment 10

[Christophe Bonjean]

We deployed the validation of deliverability process on February 3, 2025.

This concludes the identified remedial activities.

Incident Report Closure Summary

• Incident Description:

Two certificate problem reports were not delivered to GlobalSign after being recognized as malware and blocked by filtering settings. This caused a delay in the revocation of the certificates that were linked to the reported compromised keys.

• Incident Root Cause(s):

The extension used in the Certificate Problem Reports is commonly recognized as malware and is included in the default attachment filter of our email provider’s anti-malware policy. The emails were therefore not received by our report abuse team. A non-deliverability report was generated, but not delivered. When reviewing whether we were affected by the same issues as the other Bugzilla tickets, we tested and confirmed that keys and attachments of previously reported types could be received. We did not test the specific .xz extension, and did not review the non-delivered emails to the report-abuse mailbox.

• Remediation Description:

We reviewed the list of blocked file types to identify any other file type that might potentially be used to report / transmit a collection of compromised keys and removed relevant file types from the filter. We also updated the email configuration for non-delivery reports.

• Commitment Summary:

Our Certificate Problem Report deliverability is now subject to validation on a quarterly basis.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 11

[Ben Wilson]

I'll close this on or about Friday, 7-Feb-2025.