1687330 - E-Tugra: Intermittent OCSP response with status 'Unknown'

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Matthias]

Attachment: Screenshot from 2021-01-18 17-27-25.png

The OCSP responder for E-Tugra responded with status 'Unknown' for https://crt.sh/?id=3846647699&opt=ocsp. It is my understanding that that response-status is not a correct response status for certificates issued by the CA.

I have tried to get such a response through openSSL, but have failed to replicate this in the command-line. However, I have seen this happen in the browser.

Comment 1

[Matthias]

Attachment: Screenshot from 2021-01-18 17-27-50.png

Comment 2

[Matthias]

Attachment: Screenshot from 2021-01-18 17-56-08.png

Second error; note the new 'last checked' date

Comment 3

[Davut Tokgöz]

This is a preliminary report.
2. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On 18 Jan 2021 16:55 (UTC), This bug is created by Matias and was assigned to us.
4. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
19 Jan 2021 05:00 (GMT) We get the bug and begin to work.
19 Jan 2021 16:00 (GMT) The problem was fixed. We investigate the problem is related with the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1687139 . So we began detailed investigation. And also we began to search all system to find other certificates that affected.
5. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
This is still under investigation.
5. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
We began to search all system to find other certificates that affected.
6. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
We are searching it. Until now we did not find.
7. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
This is still under investigation.
8. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
This is pending completion of our investigation.

Comment 4

[Davut Tokgöz]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   On 18 Jan 2021 16:55 (UTC), This bug is created by Matias and was assigned to us.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   19 Jan 2021 05:00 (GMT) We get the bug and begin to work.
   19 Jan 2021 16:00 (GMT) The problem was fixed. We investigate the problem is related with the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1687139 . So we began detailed investigation. And also, we began to search all system to find other certificates that affected.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   There is no problem on issuing certificates or other CA functionalities
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   For the mentioned certificate, the ocsp responses was Intermittently with status unknow.
5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   No more certificates had the same problem on ocsp response.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   Etugra has many ocsp server with run with global load balancing. All new certificates and certificate status changes are published to server and replicated. We find out that the mentioned certificates are not published one of the ocsp servers and naturally some responses of this certificates were got as ‘unknown’. An error was throwed by the publisher system, but we saw that this error format was not included in our SIEM systems to be able generate an alarm.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

• Our SIEM alarms rules were upgraded with any kind of error in the Certificate issuing system
• A new component are being developed in Quality Control module that run on every certificate issue known as a part of post validation controls. The component will check all the changes of a certificates on all ocsp server after any changes happened on a certificate. It is being integrated with our SIEM systems. We will complete and publish it at the end of this week.

Comment 5

[Ryan Sleevi]

Do you have an update? Did you complete it at the end of the week, as expected?

Comment 6

[Davut Tokgöz]

Update and Close Request
A new component was developed in Quality Control module that run on every certificate issue and every status change on any certificate. The component is checking whether ocsp results is compatible with the latest status of the certificate. It was integrated with our SIEM systems. We completed and put it production 12th, February.

Comment 7

[Ben Wilson]

I believe that this incident can be closed and will schedule it for closure on Wed. 7-Apr-2021.