1855997 - SHECA: CRLs unavailable, not downloading

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ben Wilson]

According to CRL Watch (https://sslmate.com/labs/crl_watch/), there are errors with the following SHECA CRLs:
231702 https://httpd.sslmate.com/labs/issuer_info/231702
Shanghai Electronic Certification Authority 2023-09-29T14:48:33+00:00 error parsing http://crl.global.sheca.com/trustasiarsaovtlscas1.crl: should have been updated by 2023-09-29 04:03:55 +0000 UTC
266271 https://httpd.sslmate.com/labs/issuer_info/266271
Shanghai Electronic Certification Authority 2023-09-29T14:49:10+00:00 error parsing http://crl.global.sheca.com/xinnetovssl.crl: should have been updated by 2023-09-29 04:03:55 +0000 UTC
29545 https://httpd.sslmate.com/labs/issuer_info/29545
Shanghai Electronic Certification Authority 2023-09-29T14:46:08+00:00 HTTP error from http://crl.global.sheca.com/evssl-1.crl: 403 Forbidden
29549 https://httpd.sslmate.com/labs/issuer_info/29549
Shanghai Electronic Certification Authority 2023-09-29T14:46:08+00:00 HTTP error from http://crl.global.sheca.com/sslg3-1.crl: 403 Forbidden

Comment 1

[jasmine.tang]

(In reply to Ben Wilson from comment #0)

According to CRL Watch (https://sslmate.com/labs/crl_watch/), there are errors with the following SHECA CRLs:
231702 https://httpd.sslmate.com/labs/issuer_info/231702
Shanghai Electronic Certification Authority 2023-09-29T14:48:33+00:00 error parsing http://crl.global.sheca.com/trustasiarsaovtlscas1.crl: should have been updated by 2023-09-29 04:03:55 +0000 UTC
266271 https://httpd.sslmate.com/labs/issuer_info/266271
Shanghai Electronic Certification Authority 2023-09-29T14:49:10+00:00 error parsing http://crl.global.sheca.com/xinnetovssl.crl: should have been updated by 2023-09-29 04:03:55 +0000 UTC
29545 https://httpd.sslmate.com/labs/issuer_info/29545
Shanghai Electronic Certification Authority 2023-09-29T14:46:08+00:00 HTTP error from http://crl.global.sheca.com/evssl-1.crl: 403 Forbidden
29549 https://httpd.sslmate.com/labs/issuer_info/29549
Shanghai Electronic Certification Authority 2023-09-29T14:46:08+00:00 HTTP error from http://crl.global.sheca.com/sslg3-1.crl: 403 Forbidden

Hi Ben,We are aware of this bug and are urgently repairing it. We will provide a complete incident report within a week.Thank you.

Comment 2

[jasmine.tang]

Incident report：

1.How your CA first became aware of the problem (e.g., via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP or CCADB public mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

On October 1, 2023, SHECA learned through bugzilla that there was a problem with the crl address.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.

2023-10-01 02:32 UTC SHECA learned through bugzilla that there was a problem with the crl address.
2023-10-01 04:30 UTC SHECA quickly recovered by shortening the CDN cache time and increasing the CRL validity period.
2023-10-06 10:00 UTC SHECA investigated the cause of the problem.
2023-10-06 14:00 UTC SHECA develops a solution.
2023-10-06 16:00 UTC SHECA synchronizes the system time of the scheduled task scheduling service to ensure that the CRL publishing program is executed in the planned order.

3.Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

N/A

4.In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.

N/A

5.In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list “https://crt.sh/?sha256=[sha256-hash]”, unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

N/A

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

SHECA uses two different sets of CRL issuance procedures to issue full CRLs (Full CRLs) and partitioned CRLs (Partitioned CRLs) respectively. In order to improve the maintainability of the code, we have redeveloped a publishing program that supports both full and partitioned CRL to eliminate the original CRL publishing program. During the upgrade process, in order to ensure the reliability of the full CRL, we did not directly deactivate the original full CRL issuance program. Instead, we let the new full CRL issuance program complete the issuance task first, and then the original full CRL issuance program performed the full CRL issuance process. Sign off the task, overwrite the file and push the file to the object storage server. Because the partitioned CRLs (crl.sheca.com and ldap2.sheca.com) do not provide CDN acceleration services, our new CRL publishing procedure sets a shorter CRL validity period. Unfortunately, due to the server time out-of-synchronization problem of the scheduled task scheduling service provided by the new release program, the scheduled task was executed later than planned. As a result, the CRL file issued by the original full CRL issuance program was overwritten. Because the full CRL (crl.global.sheca.com) provides CDN acceleration service, the caching strategy causes users to access expired CRL files with a certain probability.

7.List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

In response to this situation, we have made the following fixes:

7.1 Check all scheduled task scheduling services and ensure that the NTP service is enabled to synchronize system time.
Completed，2023-10-7

7.2 Our previous monitoring system monitored the source address of crl, but not the cdn address of crl. We will adjust the strategy of the monitoring system to monitor and alarm the cdn address of crl. Later, we can realize self-discovery and timely repair.
Processing，2023-10-24

Comment 3

[jasmine.tang]

As mentioned in incident report 7.2, we are proceeding as planned to adjust the strategy of the monitoring system to monitor and alarm the cdn address of crl.We will provide the latest adjustment progress on October 24th.

Comment 4

[jasmine.tang]

Hi Ben,thank you for your attention.The renovation of the CRL monitoring module has been developed and is currently in the testing phase. It is expected to be released to production environment on November 2nd. We will expedite the progress, and once it is released, we will inform you promptly.

Comment 5

[jasmine.tang]

Attachment: alarm screenshot.png

We have successfully implemented the CRL monitoring function in the production environment, allowing us to detect any abnormalities in CRL status within a remarkable 10-second timeframe. Once an abnormality is identified, our dedicated developers promptly address and resolve the issue within a maximum of 10 minutes. Please note that the provided alarm screenshot is for demonstration purposes only.

Comment 6

[jasmine.tang]

Hi Ben，we have nothing else pending to correct on our side regarding this issue and considered it close.Thank you.

Comment 7

[Ben Wilson]

I will close this on Wed. 22-Nov-2023 unless there are additional questions to address.