Closed Bug 1889217 Opened 6 months ago Closed 5 months ago

Entrust: CRL non-conformance with the TLS BRs

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ryandickson, Assigned: bruce.morton)

Details

(Whiteboard: [ca-compliance] [crl-failure] [external])

Attachments

(2 files)

qtsp1.crl.txt
2.32 KB, text/plain
Details
siemensca.crl.txt
2.20 KB, text/plain
Details

Issue: Two (2) of Entrust’s CRLs disclosed to the CCADB appear in violation of the TLS BRs by extension of violating RFC 5280. Specifically, these CRLs contain the revoked certificates field, however no revoked certificates appear present.

Relevant Policy Language:

  • RFC 5280: Section 5.1.2.6 states, “When there are no revoked certificates, the revoked certificates list MUST be absent.”

  • TLS BRs (v2.02): Section 7.2 states, “Effective 2024-03-15, the CA SHALL issue CRLs in accordance with the profile specified in these Requirements. If the CA asserts compliance with these Baseline Requirements, all CRLs that it issues MUST comply with the following CRL profile, which incorporates, and is derived from RFC 5280. Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”

How was this detected?: pkilint (and manual investigation). The issue was also corroborated with OpenSSL (examples below):

Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: /C=US/O=CrowdStrike, Inc./CN=CrowdStrike TLS CA 2022
        Last Update: Apr  2 11:05:14 2024 GMT
        Next Update: Apr  9 11:05:13 2024 GMT
        CRL extensions:
            X509v3 Authority Key Identifier: 
                keyid:55:EA:A7:45:B9:9A:F7:B6:71:31:1A:31:DF:A1:76:FE:76:92:99:7A

            X509v3 CRL Number: 
                1009
            2.5.29.60: 
                ..20240331110514Z
No Revoked Certificates.
    Signature Algorithm: sha256WithRSAEncryption
         a3:88:ba:24:15:ad:23:20:63:70:65:e3:4b:6e:4d:7e:c0:d1:
         ee:b3:0f:04:3e:32:11:07:e2:1c:e3:78:88:a9:40:01:bd:ce:
         4b:68:fb:a2:da:82:28:17:6b:a2:a6:de:4e:1c:da:3c:5e:ed:
         ca:87:9c:b6:0f:9b:9a:0a:32:15:ff:a4:90:da:05:b8:c2:38:
         0e:90:7b:e2:d0:ec:79:59:76:fb:eb:dd:48:7d:33:f3:0d:fe:
         48:a2:13:1c:87:f6:9f:a3:7d:d8:e7:ff:73:b3:6c:c2:ab:a2:
         e7:6a:cc:31:b2:d2:cb:86:23:b3:bf:55:28:da:c1:8a:36:1e:
         b9:99:96:ce:49:8c:c5:b6:f9:4c:9b:56:9a:61:ca:07:f0:fa:
         2f:11:7e:02:85:8c:25:88:30:fd:08:24:a2:f4:f0:85:63:92:
         ff:37:4b:c5:c9:f4:2f:0b:74:d9:8d:f2:08:04:d7:a1:97:58:
         26:c0:30:d2:be:74:79:c9:54:f3:11:4b:16:00:8c:ea:32:bb:
         37:1b:9e:b7:e5:43:e5:6e:2a:28:ca:e4:f0:7b:a1:7d:36:9e:
         9c:cc:2c:36:c9:e3:e9:73:c2:0d:66:54:d8:69:af:8e:17:3e:
         51:b1:78:45:4a:a4:09:e5:d0:18:e1:c7:d8:64:7c:9d:f0:58:
         ea:44:9c:41
-----BEGIN X509 CRL-----
MIIB4jCBywIBATANBgkqhkiG9w0BAQsFADBLMQswCQYDVQQGEwJVUzEaMBgGA1UE
ChMRQ3Jvd2RTdHJpa2UsIEluYy4xIDAeBgNVBAMTF0Nyb3dkU3RyaWtlIFRMUyBD
QSAyMDIyFw0yNDA0MDIxMTA1MTRaFw0yNDA0MDkxMTA1MTNaMACgSjBIMB8GA1Ud
IwQYMBaAFFXqp0W5mve2cTEaMd+hdv52kpl6MAsGA1UdFAQEAgID8TAYBgNVHTwE
ERgPMjAyNDAzMzExMTA1MTRaMA0GCSqGSIb3DQEBCwUAA4IBAQCjiLokFa0jIGNw
ZeNLbk1+wNHusw8EPjIRB+Ic43iIqUABvc5LaPui2oIoF2uipt5OHNo8Xu3Kh5y2
D5uaCjIV/6SQ2gW4wjgOkHvi0Ox5WXb7691IfTPzDf5IohMch/afo33Y5/9zs2zC
q6LnaswxstLLhiOzv1Uo2sGKNh65mZbOSYzFtvlMm1aaYcoH8PovEX4ChYwliDD9
CCSi9PCFY5L/N0vFyfQvC3TZjfIIBNehl1gmwDDSvnR5yVTzEUsWAIzqMrs3G563
5UPlbiooyuTwe6F9Np6czCw2yePpc8INZlTYaa+OFz5RsXhFSqQJ5dAY4cfYZHyd
8FjqRJxB
-----END X509 CRL-----

and...

Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: /CN=Entrust Certification Authority - QTSP1/2.5.4.97=VATES-B81188047/O=Entrust Datacard Europe S.L./C=ES
        Last Update: Apr  2 17:21:14 2024 GMT
        Next Update: Apr  9 17:21:13 2024 GMT
        CRL extensions:
            X509v3 Authority Key Identifier: 
                keyid:1C:AD:3F:9C:D7:2D:22:19:A1:9C:4B:E9:DA:F1:2A:33:F7:FB:BA:0D

            X509v3 CRL Number: 
                3864
            2.5.29.60: 
                ..20240331172114Z
No Revoked Certificates.
    Signature Algorithm: sha256WithRSAEncryption
         49:b6:cd:1c:2f:94:54:e9:e8:6e:4e:59:d5:0d:35:15:2a:ac:
         a9:ec:d0:63:a1:b8:c9:e3:af:94:15:54:99:31:34:8b:65:4e:
         42:bb:15:af:4a:7c:ff:0d:2c:b8:89:09:62:e8:5d:8e:51:16:
         be:fe:1a:d4:6c:85:4d:98:65:57:8c:39:ff:9e:2c:69:90:20:
         53:35:d8:05:1a:23:53:37:c2:38:4c:56:9b:12:bb:fd:3c:b3:
         c9:a6:8f:5d:f3:2c:17:c1:d5:b4:8c:11:86:10:e6:e4:99:1d:
         01:a2:94:d0:52:36:ed:c7:ce:ae:46:56:af:a1:d1:4d:43:a3:
         22:bb:52:a6:1c:bb:50:79:19:a1:53:14:80:34:1b:c9:cc:70:
         85:b8:ec:53:6e:b9:35:06:ef:7f:3f:07:83:0a:dd:4e:2d:e7:
         30:29:d3:65:fc:a6:d3:00:63:bd:9c:17:b6:67:3d:c4:d7:e3:
         d0:12:83:77:35:29:1a:6e:ec:21:30:04:6c:49:f6:b5:f9:10:
         4d:bb:76:99:c2:b8:2d:15:bb:60:0e:99:a6:6c:fa:ec:cc:30:
         ed:c3:11:65:61:04:48:20:2b:d4:cd:48:97:f9:6d:54:32:49:
         4a:35:c5:4e:9e:f3:91:a9:e9:59:06:1b:c5:10:3a:5d:c5:2f:
         4d:bc:a4:1c
-----BEGIN X509 CRL-----
MIICGTCCAQECAQEwDQYJKoZIhvcNAQELBQAwgYAxMDAuBgNVBAMMJ0VudHJ1c3Qg
Q2VydGlmaWNhdGlvbiBBdXRob3JpdHkgLSBRVFNQMTEYMBYGA1UEYQwPVkFURVMt
QjgxMTg4MDQ3MSUwIwYDVQQKDBxFbnRydXN0IERhdGFjYXJkIEV1cm9wZSBTLkwu
MQswCQYDVQQGEwJFUxcNMjQwNDAyMTcyMTE0WhcNMjQwNDA5MTcyMTEzWjAAoEow
SDAfBgNVHSMEGDAWgBQcrT+c1y0iGaGcS+na8Soz9/u6DTALBgNVHRQEBAICDxgw
GAYDVR08BBEYDzIwMjQwMzMxMTcyMTE0WjANBgkqhkiG9w0BAQsFAAOCAQEASbbN
HC+UVOnobk5Z1Q01FSqsqezQY6G4yeOvlBVUmTE0i2VOQrsVr0p8/w0suIkJYuhd
jlEWvv4a1GyFTZhlV4w5/54saZAgUzXYBRojUzfCOExWmxK7/TyzyaaPXfMsF8HV
tIwRhhDm5JkdAaKU0FI27cfOrkZWr6HRTUOjIrtSphy7UHkZoVMUgDQbycxwhbjs
U265NQbvfz8HgwrdTi3nMCnTZfym0wBjvZwXtmc9xNfj0BKDdzUpGm7sITAEbEn2
tfkQTbt2mcK4LRW7YA6Zpmz67Mww7cMRZWEESCAr1M1Il/ltVDJJSjXFTp7zkanp
WQYbxRA6XcUvTbykHA==
-----END X509 CRL-----

URLs identified:

Note: Other CRLs disclosed to the CCADB appear to also serve an empty set of revoked certificates, but satisfy non-TLS use cases or are issued by root CA certificates before the updated TLS BR profile requirements took effect. It's also possible other CRLs issued by Entrust will exhibit this same failure mode in the future as certificates fall off CRLs after expiry.

Assignee: nobody → bruce.morton
Status: NEW → ASSIGNED

(In reply to Ryan Dickson from comment #0)

Issue: Two (2) of Entrust’s CRLs disclosed to the CCADB appear in violation of the TLS BRs by extension of violating RFC 5280. Specifically, these CRLs contain the revoked certificates field, however no revoked certificates appear present.

Hi Ryan, thanks for the notice. We are investigating will provide an incident report.

Investigation indicates the issue was bug in our CRL generation system which did not meet the RFC 5280 section 5.1.2.6 requirement. This issue would impact any CRL which did not have a revoked certificate.

Today the CRL generation software was updated for the subordinate CA online system and the root CA offline system. All root CA CRLs were issued and published. All non-compliant Subordinate TLS CA CRLs issued and published.

We are working on our incident report, which we plan to post next week.

Incident Report

Summary

Entrust developed and deployed a new system to provide CRLs and OCSP responses for our certificate service in 2019. The new CRL generation software did not support Section 5.1.2.6 of RFC 5280: which states, “When there are no revoked certificates, the revoked certificates list MUST be absent.”

In July 2023, the CA/Browser Forum approved Ballot SC-063 v4: Make OCSP Optional, Require CRLs, and Incentivize Automation. The ballot states “Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”

Linting software pki.lint was updated to lint the CRLs and provided a test for RFC 5280 section 5.1.2.6. The Entrust CRLs were manually tested and this incident was opened on 2 April 2024.

On 5 April 2024, updated CRL generation software was installed for the online system which supports subordinate CAs and the offline system which supports root CAs. All root CA CRLs were issued and published. All non-compliant Subordinate TLS CAs issued and published. Pkilint was run against all Entrust CRLs, which also found two non-TLS CA CRLs that were non RFC 5280 compliant. These CRLs automatically updated within our 12-hour CRL issue cycle and became compliant.

Impact

There is no security impact; however, Entrust has signed CRLs for many CAs which do not meet the requirements of RFC 5280 by providing a revocation list when there are no revoked certificates.

Timeline

All times are UTC.

2019-09:

  • Entrust certificates services started to migrate to a the new CRL generation software to all existing TLS roots and subordinate CAs.

2023-07-06:

  • 16:00 Ballot SC-063 v4: Make OCSP Optional, Require CRLs, and Incentivize Automation vote passed. The ballot states “Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”

2024-03-15:

  • 16:00 Ballot SC-063 became effective.

2024-04-02

  • 19:39 This Bugzilla was posted to provide notice of CRL issue.
  • 20:20 Entrust investigated the issue and notified the development team that there an issue with CRL generation software.
  • 20:54 Entrust tested the TLS CA’s CRLs using pkilint and discovered additional CRLs provided in the Appendix with a revocation list and no revoked certificates. It was concluded the issue is CA independent and applies to the CRL generation software.

2024-04-03

  • 19:47 Entrust developed an update to the CRL generation software for testing. The tests were successful.

2024-04-05

  • 15:10 CRL Generation Software with empty revocation list fix deployed to production.
  • 16:42 New fixed CRLs issued and published. (aftov1ca.crl, crowdstrikeca.crl, qtsp1.crl, and siemensca.crl)

Root Cause Analysis

The CRL generation software was created using the PHASNoo Go library. There was a bug in the original PHASNoo Go library: Go fixed in 1.15, but instead of fixing the existing implementation, anew one was created to keep the "compatibility contract".

We had forked PHASNoo after the fix was released but before the deprecation notice and so continued to use the faulty implementation.

Lessons Learned

What went well

  • The bug was fixed and validated within the expected time frame.

What didn't go well

  • We have a handful of forked packages and we know that dependency tools (WhiteSource/Mend) do not work on them. However, we did not maintain a list of forked packages/repos to check for updates/alerts or revisit if we still need a forked dependency .

Where we got lucky

  • the bug has no security implications
  • the bug has no known interoperability implications

Action Items

Action Item Kind Due Date
Run pkilint against all CRLs Detection DONE
Update CRL generation software installed for online and offline CRL systems. CRLs issued and published Correction DONE
Update automated test to cover the added requirement Prevention DONE
Set up plan for periodically reviewing forked libraries release note patches Detection DONE

Appendix

See CRLs in attachments.

Attached file qtsp1.crl.txt —
Attached file siemensca.crl.txt —

Comment on 1890685

I keep seeing this type of inappropriate information in the impact section of incident reports. The guidelines at https://www.ccadb.org/cas/incident-report#incident-reports are clear. This section isn't for arguing that the incident isn't a big deal. You should be describing the amount and nature of the certificates that should be revoked: how many certificates are affected? When were they issued? When do they expire? How are they related? (EV and OV, EV-only, etc.)

Same thing applies here.

The Impact section should contain a short description of the size and nature of the incident. For example: how many certificates, OCSP responses, or CRLs were affected; whether the affected objects share features (such as issuance time, signature algorithm, or validation type); and whether the CA Owner had to cease issuance during the incident.

Please apply the learnings from your other threads to these incidents.

Do you know how many CRLs were impacted?

PHASNoo Go library.

Is this open source? If so, can you link this?

Also, am I understanding this was for Go, as in Go the programming Language? I'm confused, did you go out of your way to not use the standard library here?

(In reply to amir from comment #7)

PHASNoo Go library.

Is this open source? If so, can you link this?

Sorry for the confusion, the “PHASNoo Go library” is an internal library based on crypto/x509.

Also, am I understanding this was for Go, as in Go the programming Language? I'm confused, did you go out of your way to not use the standard library here?

We have an internal crypto/x509 fork that gets regular updates from the upstream, but we used the CreateCRL function instead of the RFC 5280 compliant CreateRevocationList function that was added in Go 1.15 (released 2020-08-11) were we overlooked the release note message, also the release notes of Go 1.19 (released 2022-08-02) did not say anything about the deprecation of CreateCRL, which only mentioned the deprecation of the parse functions.

(In reply to Bruce Morton from comment #3)

2023-07-06:

  • 16:00 Ballot SC-063 v4: Make OCSP Optional, Require CRLs, and Incentivize Automation vote passed. The ballot states “Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”

2024-03-15:

  • 16:00 Ballot SC-063 became effective.

Did Entrust do any reviews or make any system changes during this time? How does Entrust determine if changes to systems are required after a ballot passes? What steps does Entrust take to verify that systems and processes comply wit the BRs?

The report should include an analysis of the failure to detect this noncompliance until it was reported by a third party. This incident, like bug 1879602, seems to be a failure of keeping track of requirements, analyzing how systems are affected by changing requirements, and reconfirming compliance with requirements on a regular basis. The report includes a prevention item for this specific CRL issue, but it doesn't include action items to ensure that similar incidents do not reoccur.

(In reply to Mathew Hodson from comment #9)

(In reply to Bruce Morton from comment #3)

2023-07-06:

  • 16:00 Ballot SC-063 v4: Make OCSP Optional, Require CRLs, and Incentivize Automation vote passed. The ballot states “Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”

2024-03-15:

  • 16:00 Ballot SC-063 became effective.

Did Entrust do any reviews or make any system changes during this time? How does Entrust determine if changes to systems are required after a ballot passes? What steps does Entrust take to verify that systems and processes comply with the BRs?

This incident was not caused by changed requirements but to RFC non-compliance, see the 5 Whys below for clarity.

1. Why was there a problem?
We issued RFC 5280 non-compliant CRLs.

2. Why were these CRLs not RFC compliant?
These CRLs included an empty list of revoked certificates while RFC 5280 section 5.1.2.6 states:

When there are no revoked certificates, the revoked certificates list MUST be absent.

3. Why was an empty list of revoked certificates included?
We used the CreateCRL function in Go which was non-RFC complaint.

4. Why did we use a non-compliant RFC implementation?
We overlooked that the CreateRevocationList function was added in Go 1.15 (released 2020-08-11), and the release note message about this new RFC compliant function. When the CreateCRL function was deprecated in Go 1.19 (released 2022-08-02) the release notes did not say anything about this deprecation or the RFC non-compliance.

5. Why did we not detect this issue earlier?
We had no test for this type of issue and had no reason to assume that this core Go function would be RFC non-compliant. In addition, most certificate viewers will show an empty list of revoked certificates even if the list is absent. However, we could have detected this issue with pkilint, which was not available at the time this code was written but could have been used to check for any potential issues.

The report should include an analysis of the failure to detect this noncompliance until it was reported by a third party. This incident, like bug 1879602, seems to be a failure of keeping track of requirements, analyzing how systems are affected by changing requirements, and reconfirming compliance with requirements on a regular basis. The report includes a prevention item for this specific CRL issue, but it doesn't include action items to ensure that similar incidents do not reoccur.

Please note that these incidents did not occur due to not keeping track of the requirements.

We do the following to track requirement changes:

  • Upcoming and new ballot status is discussed on a weekly team meeting

  • Final ballots are review by our product management, development, operations, and compliance teams to ensure understanding of the ballot requirements and ownership of implementation

  • Development tracks ballot implementation through release plans

  • Operations tracks ballot implementation and their process will be updated based on https://bugzilla.mozilla.org/show_bug.cgi?id=1879602#c10.

To reconfirm compliance we do the following:

  • Pre-issuance and post-issuance certificate linting

  • Monitor the PKI which now also includes CRL Watch and OCSP Watch

We are open to review your suggestions as to what should be included in processes.

All actions are complete, there are no updates for this week and we will continue to monitor the bug.

(In reply to Bruce Morton from comment #10)

To reconfirm compliance we do the following:

  • Pre-issuance and post-issuance certificate linting

  • Monitor the PKI which now also includes CRL Watch and OCSP Watch

We are open to review your suggestions as to what should be included in processes.
To confirm CRL Watch and OCSP Watch were added to your environment in response to this issue? What were your practices for detecting any CRL or OCSP issue outside of the specific one highlighted here?

(In reply to Wayne from comment #12)

To confirm CRL Watch and OCSP Watch were added to your environment in response to this issue? What were your practices for detecting any CRL or OCSP issue outside of the specific one highlighted here?

Yes, CRL Watch and OCSP Watch were added based on this issue. Prior practices were to review the CRL or OCSP response with OpenSSL; then we would add monitoring for specific items. We would also use https://certificate.revocationcheck.com/ to test CRL and OCSP, but this was not used as a monitoring practice.

I intend to close this bug on Friday, 3-May-2024.

Flags: needinfo?(bwilson)

There are no updates for this week and we will continue to monitor the bug.

Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED

(I've closed this, but it will still appear on the list of Entrust compliance issues being written.)

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: