1908690 - Sectigo: Temporary unavailability for subset of CRLs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Incident Report

Summary

On July 16th, 2024 at approximately 18:40 UTC, we made a change to our infrastructure as the final stage of decommissioning a deprecated CRL distribution system. This change resulted in 13 CRLs becoming unavailable and responding with a 504 HTTP error code (Gateway Time-out).

Impact

13 CRLs responded with a 504 Gateway Time-out error rather than responding with the respective CRL, between 18:44 (first detected) and approximately 19:53 on July 16th, 2024.

Timeline

All times are UTC.

2024-03-18:

• 08:12 We create a ticket to start the switchover process for moving CRL distribution responsibilities to our new CertStatus system as previously introduced in bug 1741777.

2024-03-20:

• 15:14 We start processing the ticket.
• 15:49 We change our CDN settings to point our CRL domains towards the new system as origin address.

2024-03-22:

• 13:50 We request the now thought to be deprecated system to be decommissioned. After this moment, the request stalls.

2024-04-23:

• 12:57 An internal message is sent, requesting a confirmation that all of our CRLs are now served from the new CertStatus platform and that no requests are now being served by the now deprecated platform. Discussion continues, final action ultimately stalls.

2024-07-16:

• 09:30 We escalate the original ticket to be processed as soon as possible.
• 18:40 A change is made to our infrastructure, ceasing use of the deprecated CRL publishing system.
• 19:25 Automated internal monitoring reports a small number of CRLs are returning a 504 Gateway Time-out error.
• 19:31 We confirm the CRLs are indeed returning a 504 HTTP Error code. We escalate the issue internally.
• 19:36 Our Infrastructure and DevOps team confirm they are investigating.
• 19:53 The change made earlier in the day is reversed, restoring the availability of the affected CRLs.

Root Cause Analysis

The original ticket stalled when other tasks were prioritized.
When the ticket was once again picked up this month, this was done by a different engineer. The engineer interpreted the request to “decommission” the deprecated system as a request to turn the services off.

Rather, the intent was, as was also clarified in internal messages back in April, that confirmation should be provided that indeed no CRLs were being served by the deprecated system, prior to it being disabled.

Further investigation shows that the affected CRL hostnames did not yet route through our CDN, and instead directly served CRLs from the deprecated system. Note that, while the system was deprecated, it did serve the correct CRLs.

Lessons Learned

What went well

• The issue was detected by our internal monitoring and remediated in relatively short time.

What didn't go well

• We did not review the server logs to confirm that no CRLs were being served from the deprecated system.
• We were unaware the three affected hostnames were still routed directly to our infrastructure, rather than through our CDN.

Where we got lucky

• The issue only affected a small subset of all our CRLs.

Action Items

Action Item	Kind	Due Date
Review and adjust policies for the deprecation of legacy systems.	Prevent	2023-08-15
Review DNS settings for our domains that are used for OCSP, CRT and CRL services and update where needed.	Prevent	2023-08-15

Appendix

URLs of affected CRLs

• http://crl.securecore-ca.com/SecureCoreRSACodeSigningCA.crl
• http://crl.securecore-ca.com/SecureCoreRSAEVCA.crl
• http://crl.securecore-ca.com/SecureCoreRSAOVCA.crl
• http://crl.securecore-ca.com/SecureCoreRSADVCA.crl
• http://crl.securecore-ca.com/SecureCoreRSASecureClientandEmailCA.crl
• http://crl.incommon-rsa.org/InCommonRSAStandardAssuranceClientCA.crl
• http://crl.incommon-rsa.org/InCommonRSAServerCA.crl
• http://crl.incommon-rsa.org/InCommonRSACodeSigningCA.crl
• http://crl.trustasiassl.com/TrustAsiaRSASecureClientandEmailCA.crl
• http://crl.trustasiassl.com/TrustAsiaRSACodeSigningCA.crl
• http://crl.trustasiassl.com/TrustAsiaRSADVSSLServerCA.crl
• http://crl.trustasiassl.com/TrustAsiaRSAOVSSLServerCA.crl
• http://crl.trustasiassl.com/TrustAsiaRSAEVSSLServerCA.crl

Comment 1

[Martijn Katerbarg]

Ben, based on our action items, we would like to request a next-update for 2024-08-15

Comment 2

[Martijn Katerbarg]

Both action items shown in comment 0 have been completed. We are monitoring this bug for any questions and/or comments.

After further internal discussions relating to the first action item, we are adding a third action item to this incident. Updated table below:

Action Item	Kind	Due Date
Review and adjust policies for the deprecation of legacy systems	Prevent	Completed
Review DNS settings for our domains that are used for OCSP, CRT and CRL services and update where needed	Prevent	Completed
Switch off our deprecated CRL distribution platform, having confirmed that it is no longer receiving any requests	Prevent	Completed

Comment 3

[Martijn Katerbarg]

As all action items have been completed and no questions have been raised, we would like to request closing this bug.

Comment 4

[Ben Wilson]

I will look at closing this on Friday, 23-Aug-2024.