1931886 - Entrust: CRL missing revocation reasonCode

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Bruce Morton]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36

Comment 1

[Bruce Morton]

Preliminary Incident Report

Summary

Based on evaluating a monitoring alert, it was determined the CRL for our TLS CAs did not provide the reasonCode for certificates which were not revoked for the unspecified reason. The TLS BRs section 7.2.2 states:

MUST be present unless the CRL entry is for a Certificate not technically capable of causing issuance and either 1) the CRL entry is for a Subscriber Certificate subject to these Requirements revoked prior to July 15, 2023 or 2) the reason for revocation (i.e., reasonCode) is unspecified (0).

The error was introduced based on a recent software update. The update was rolled back and all CRLs for TLS CAs were reissued.

Impact

The revocation reasonCode was missing for CRLs subordinate CAs including the TLS subordinate CAs. The CRLs have been corrected.

Next steps

A full incident report including the root cause analysis, lessons learned, and action items will be posted per the CCADB full incident report requirement.

Comment 2

[Bruce Morton]

Incident Report

Summary

Based on evaluating a monitoring alert, it was determined the CRL for our TLS CAs did not provide the reasonCode for certificates which were revoked with a reason other than “unspecified reason”. The TLS BRs section 7.2.2 states:

reasonCode

When present (OID 2.5.29.21), MUST NOT be marked critical and MUST indicate the most appropriate reason for revocation of the Certificate.

MUST be present unless the CRL entry is for a Certificate not technically capable of causing issuance and either 1) the CRL entry is for a Subscriber Certificate subject to these Requirements revoked prior to July 15, 2023 or 2) the reason for revocation (i.e., reasonCode) is unspecified (0).

The error was introduced due to a recent software update. The update was rolled back and all CRLs for TLS CAs were reissued.

Impact

The revocation reasonCode was missing in CRLs for subordinate CAs including the TLS subordinate CAs for certificates that were revoked with a reason other than “unspecified reason”. The CRLs have been corrected.

OCSP responses were not impacted.

Timeline

All times are UTC.

2024-11-12:

• 18:19 A software update was deployed into production that included library updates.

2024-11-13:

• 11:38 A non–critical alert was received of a possible discrepancy between the revocation status of a single certificate in our portal database compared to that in our OCSP/CRL system. The certificate was issued from a private CA (non-publicly trusted). After investigation, nothing abnormal was discovered with the certificate or CRL, and this was treated as a false-positive alert.

2024-11-14:

• 11:38 A non–critical alert was received of another possible discrepancy between the revocation status of a single certificate in our portal database compared to that in our OCSP/CRL system. The certificate was issued from a publicly trusted TLS CA.
• 16:34 Investigation of the alert revealed that the reasonCode was missing from the CRL. The most recent software update was suspected as the cause.
• 17:56 It was confirmed with development that a bug was introduced with the library updates deployed on 2024-11-12.
• 18:34 The correct roll back version was identified. The release was rolled back and affected CRLs were re-issued.

Root Cause Analysis

Why was the reasonCode not included in the CRL?

The CRL generation code was updated to use the Golang x509.CreateRevocationList API. We previously used a modified version of x509.Certificate.CreateCRL that has now been deprecated. We updated the CRL generation code to minimize the code that diverges from the CRL generation functions in the Go standard library.

While we have unit tests to check the contents of CRLs, these tests were executed without problems after the code update, so no issue with the contents of the CRLs was detected.

Why didn't the unit tests detect that the reasonCode was not in the CRL?

Unit test code was written using a mocked version of the function that generates the CRL entry extensions. The unit test was not testing the same function used in the production code to generate the extensions. This was a bug in the unit test.

The unit test has been updated to use the same functions as the production code.

Why was the issue not found through other compensating controls?

We have two other direct controls in place to test the proper operation of the system: functional tests and linting.

Functional tests were not written to detect this issue. Our functional tests validate the number of CRL entries along with other parameters, but do not check for the presence or accuracy of the reasonCode extension.

We lint and monitor our published CRLs on a daily basis. The linters did not detect the missing reasonCode, since they do not have access to the full context of the revocation request submitted by the subscriber. It is valid to have no reasonCode in a CRL, if the reason specified by the subscriber was unspecified (0).

Lessons Learned

What went well

• No mis-issued certificates
• Existing monitoring and investigation process enabled detection of the problem

What didn't go well

• While all CRLs are automatically linted daily, the linters did not detect the missing reasonCode, since it is valid to have no reasonCode when the revocation reason is unspecified (0). The linters have no knowledge of the reasonCodes specified when the subscriber requests a revocation.

Where we got lucky

• We were lucky to have received alerts for an unrelated problem. Our investigation led us to discover this issue where the reasonCode was missing from the CRLs.

Action Items

Action Item	Kind	Due Date
Update all unit tests to use the same code used in the software	Prevent	Completed
Add functional test to validate CRL extension content	Prevent	Completed
Improve system monitoring to include reasonCode validation in addition to linting	Detect	2024-01-31

Appendix

Details of affected certificates

N/A

Comment 3

[Bruce Morton]

Correcting typo.

Action Items

Action Item	Kind	Due Date
Update all unit tests to use the same code used in the software	Prevent	Completed
Add functional test to validate CRL extension content	Prevent	Completed
Improve system monitoring to include reasonCode validation in addition to linting	Detect	2025-01-31

Comment 4

[Bruce Morton]

We will continue to monitor.

Comment 5

[Bruce Morton]

We will continue to monitor. Please set the next update to 2025-01-31. Thanks.

Comment 6

[Bruce Morton]

Action Items

Action Item	Kind	Due Date
Update all unit tests to use the same code used in the software	Prevent	Completed
Add functional test to validate CRL extension content	Prevent	Completed
Improve system monitoring to include reasonCode validation in addition to linting	Detect	Completed

Incident Report Closure Summary

• Incident Description: CRL for our TLS CAs did not provide the reasonCode for certificates which were not revoked for the unspecified reason.
• Incident Root Cause(s): Unit test code was written using a mocked version of the function that generates the CRL entry extensions. The unit test was not testing the same function used in the production code to generate the extensions. This was a bug in the unit test.
• Remediation Description: A functional test was added to validate CRL extension content. The system monitoring was improved to include reasonCode validation in addition to linting.
• Commitment Summary: Entrust committed to improving our testing, by addressing the problem and adding an update to all unit tests to use the same code used in the software.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 7

[Ben Wilson]

I'll close this on or about Wed. 5-Feb-2025.