1838315 - IdenTrust: Certificate with missing details flagged by OCSP Watch

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36

Steps to reproduce:

On June 7, 2023 we have noticed an IdenTrust issued certificate in the SSLmate OCSP watch monitor tool. The issue was corrected on the same day. We are gathering details and will supply a complete incident report no later than by June 24, 2023.

Comment 1

[IdenTrust]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On June 7, 2023, during routine monitoring, we detected an IdenTrust-issued certificate in the SSLMATE OCSP Watch monitoring tool that required attention. The problem was promptly resolved on the same day. Upon investigation, it was found that the flagging occurred due to a system outage, which resulted in the issuance of a precertificate without a serial number. This led to an unknown OCSP status. According to the Subscriber (Server) Certificate Profile outlined in B.R. Section 7.1.2.7, the inclusion of a serial number is mandatory.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2023-06-06 11:46:50 MST: Customer requested an OV TLS certificate via the online application
2023-06-06 12:26:14 MST: The request was authorized
2023-06-07 05:24:40 MST: Customer started the certificate retrieval process triggering precertificate issuance
2023-06-07 05:25:42 MST: System started sending publish request with profile for cert type; issuance and publishing of the precertificate
2023-06-07 05:27:42 MST: System received java exception error: Connection reset due to system outage
2023-06-07 05:27:42 MST: System restarted sending publish request with profile for cert type; issuance and publishing of pre-certificate
2023-06-07 05:29:42 MST: System received java exception error: Connection reset due to system outage
2023-06-07 09-57:00 IdenTrust noticed the certificate in the SSLMATE OCSP Watch
2023-06-07 11:39:30 MST: the certificate status was updated with the missing serial number correcting the discrepancy in the monitoring tool.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   Yes

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

One OV TLS certificate issued on 2023-06-07

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   https://crt.sh/?id=9594204263

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

During the certificate creation process, a hardware malfunction occurred, leading to resource limitations and performance issues. As a result, the applicant attempted to retrieve the certificate multiple times, inadvertently generating a precertificate without a serial number.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

To prevent the possibility of multiple retries during certificate retrieval, which may lead to the issuance of incomplete certificates, we will enhance the code by implementing a stronger locking mechanism.

We aim to implement this code update by September 30, 2023, and will provide monthly updates on our progress. The next update is scheduled for July 31, 2023.

Comment 2

[IdenTrust]

We are on track to deploy the fix to avoid recurrency of this issue. A status update will be provided by August 31, 2023.

Comment 3

[IdenTrust]

We remain on track to deploy the fix by the end of September to avoid recurrency of this issue. A status update will be provided by September 30, 2023.

Comment 4

[Phil Porada]

The crt.sh link is for a valid precertificate, but I was curious if you can attach the precertificate that was generated without the serial number. I want to see it out of morbid curiosity.

Comment 5

[IdenTrust]

During the certificate creation process, a hardware malfunction occurred, leading to resource limitations and performance issues. As a result, the applicant attempted to retrieve the certificate multiple times, inadvertently generating a precertificate with serial number overwritten in the IdenTrust database by subsequent attempt by the applicant. This led to the precertificate generated whose serial number was overwritten by subsequent retry of retrieval during system instability. The precertificate was published to the cert transparency servers (https://crt.sh/?id=9594204263).

Comment 6

[IdenTrust]

The code revision has been tested and is on track for our September 30 release date. We will provide a final update on October 2 to confirm that it is in place and is working as expected.

Comment 7

[IdenTrust]

The code was successfully deployed on 9/30/2023 and it is working as expected.
We consider this issue closed as completed.

Comment 8

[Ben Wilson]

I will close this on 11-Oct-2023 unless there are questions or concerns to still address.