1828717 - FNMT: CRL problems displayed during the monitoring

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Amaya Espinosa]

Attachment: CRL problems displayed during the monitoring.pdf

Comment 1

[Amaya Espinosa]

1.How your CA first became aware of the problem (e.g., via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP or CCADB public mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of the problem when Ben Wilson notified us that there were error with 7 of our CRLs in SSLMate’s CRL-Watch website by email on April 14 at 19:12.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.

All the date and time of following timelines are described:

• 2023/04/14 19:12: Ben notified us the problems in SSLMate’s CRL Watch website by email.
• 2023/04/14 19:30: The compliance team confirmed the finding and checked the SSLMate CRL-Watch website. At that time, the errors displayed were "i/o timeout". It was escalated to the engineering team
• 2023/04/14 19:40: The engineering team checked the status of the CRL responses and verified that they were fine, accessible and downloadable.
  Nevertheless, the response time could be longer than normal due to the DDoS attack we were suffering and mitigation measures we had activated that morning at 9:15 am.
• 2023/04/14 21:33: After analyzing the first reported errors, we believe that these are, actually, communication errors that have been misinterpreted. The engineering team rechecked that the CRLs are available and downloading correctly.
• 2023/04/15 9:00: The services were checked and they worked fine. After analyzing the traffic, it was decided to keep part of the mitigation measures activated in order to continue providing normal service. We continued to manually check SSLMate CRL-Watch the following day.
• 2023/04/16 9:11: The SSLMate CRL-Watch website is checked. At that time, errors "403 Forbidden" were detected and the engineering teams were notified.
• 2023/04/16 12:20: Our engineering team reviewed our systems and detected that the pattern of requests executed by the CRL Watch script matched the pattern involved in the DDoS attack and, for this reason, it was being blocked. Measures to mitigate the attack were maintained until Monday for security reasons.
• 2023/04/17 9:00: Our IPS completed the mitigation and removed the traffic blocks that had been set up.
• 2023/04/17 14:34: Confirmed that the errors were removed from CRL Watch.

3.Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

This issue only affected CRL responses and SSLMate’s CRL-Watch website monitoring. Certificate issuing services were not stopped.

4.In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.

No certificates were affected.

5.In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list “https://crt.sh/?sha256=[sha256-hash]”, unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

No certificates were affected.

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Issues displayed on SSLMate’s CRL Watch website between April 14 and 17 were due to DDoS attack and the mitigation measures that we put in place.

7.List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

We will add monitoring to SSLMate’s CRL Watch website to detect proactively any problem to affect us. We expect this to be done at latest by this week.

Comment 2

[Amaya Espinosa]

Monitoring at SSLMate’s CRL Watch website has been configured in our 24x7 alert system.

Comment 3

[Ben Wilson]

Unless there are items to discuss, I intend to close this on Friday, 29-Sept-2023.