1887753 - Entrust: Delay in Updating CPS

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Bruce Morton]

Attachment: 20240325-EV-TLS-Certificates-OID-CPS-finalcerts.txt

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Comment 1

[Bruce Morton]

Incident Report

Summary

As we investigated the https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 issue, we recognized that the CPS had not been updated with the Entrust EV TLS certificate policy identifier. The Entrust CP identifiers were removed for certificates issued in accordance with the TLS Baseline Requirements and EV Guidelines to meet the intent of the MAY stipulation for “Any other identifier” (see https://cabforum.org/working-groups/server/baseline-requirements/requirements/#71279-subscriber-certificate-certificate-policies).
With the Entrust EV TLS CP OID removed, we had customer cases where the certificates were not displayed as EV certificates in some browsers. We believe the reason was that the associated roots had been embedded with the EV TLS CP OID and not the EV TLS reserved CP OID. Adding the Entrust EV TLS OID back to the certificates resolved the issue; however, the CPS was not updated for this change.
In addition, the CPS was being updated to address incident https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 and https://bugzilla.mozilla.org/show_bug.cgi?id=1886467. As a result, an incomplete CPS update was posted on 21 March 2024; the complete version was provided on 22 March 2024.
The CPS updates indicate the date when we stopped using the CP OID and the date where we started using the CP OID again.

Impact

The certificates were issued in accordance with the requirements of the TLS BRs, but were not issued in accordance with the Entrust CPS. There is no security risk to relying parties or subscribers with these certificates.

Timeline

All times are UTC.

2023-04-11:

• TLS BR 2.0 released based on SC-62 certificate profiles ballot, which provided MAY stipulation for “Any other identifier”. Entrust decided to respect the MAY stipulation not knowing the impact to EV TLS certificates.
  2023-09-11:
• 17:40 Entrust EV TLS CP OID removed from EV TLS certificates.
  2023-09-22:
• 15:34 Entrust EV TLS CP OID added back to EV TLS certificates.
  2023-10-16:
• CPS update was posted stating CP OID 2.16.840.1.114028.10.1.2 would not be used.
  2024 -03-18
• Review of the certificate profile information for incident #1883843 indicated the CPS was not correct.
  2024-03-21
• Incomplete CPS update was posted. The update addressed allowing the missing CP OID, but did not address changes from other incidents which included EV CP policyQualifier and changes required for clientAuth only certificates,
  2024-03-22
• Complete CPS update was posted to address https://bugzilla.mozilla.org/show_bug.cgi?id=1883843, https://bugzilla.mozilla.org/show_bug.cgi?id=1886467 and this incident.

Root Cause Analysis

The certificate profile change procedure did not include a step to update the CPS. In addition, the team working on the incident did not consider that there could be a CPS impact.

Lessons Learned

• Flags need to be set regarding changes that must be reviewed to ensure the compliance documentation is updated as part of the change update.
• CPS clauses could be less restrictive to limit impacts due to implementation of ballots and other changes.

What went well

What didn't go well

• There was already an incident under investigation which impacted the CPS. The investigation found two other incidents including this item. Due to the pressure to coordinate the CPS updates for the incidents, it took a few days to update properly.

Where we got lucky

• Issuing not in accordance with the CPS did not provide more security exposure to relying parties and subscribers.
• Incident impacts a subset of certificates being revoked per #1883843 incident.

Action Items

Action Item	Kind	Due Date
Review and update release and emergency release process to ensure compliance documentation stays synchronized	Prevent	2024-04-30

Appendix

Details of affected certificates

See attached.

Comment 2

[Rob Stradling]

Where we got lucky
...Incident impacts a subset of certificates being revoked per #1883843 incident.

Hi Bruce. Are you implying that this "Late CPS Update" incident affects only certificates that were already due to be revoked due to bug 1883843? I think this is what you're saying, because IINM the latest notBefore timestamp of any certificate included in attachment 9393144 [details] matches the latest notBefore timestamp of any certificate being revoked due to bug 1883843. That shared latest notBefore timestamp is 2024-03-18 21:33:52 UTC.

2024-03-21
Incomplete CPS update was posted. The update addressed allowing the missing CP OID,

This CPS update appears to be https://www.entrust.com/sites/default/files/documentation/licensingandagreements/entrust-certificate-services-cps-3-17.pdf, which was published at 2024-03-21 16:34:44 UTC according to its current Last-Modified HTTP response header.

ISTM that every EV TLS certificate you issued between 2024-03-18 21:33:52 UTC and 2024-03-21 16:34:44 UTC that contains the policy OID 2.16.840.1.114028.10.1.2 needs to be added to your list of certificates affected by this "Late CPS Update" incident.

Example: https://crt.sh/?sha256=A37E57BDFE1BA0B90C13AD169300943B2DA0CAFD9437BA46600D0B3B9994B37F

Comment 3

[Bruce Morton]

Attachment: 20240326-EV-TLS-Certificates-OID-CPS-precerts.txt

Comment 4

[Bruce Morton]

Attachment: 20240326-EV-TLS-Certificates-OID-CPS-finalcerts 1.txt

Comment 5

[Bruce Morton]

(In reply to Rob Stradling from comment #2)

ISTM that every EV TLS certificate you issued between 2024-03-18 21:33:52 UTC and 2024-03-21 16:34:44 UTC that contains the policy OID 2.16.840.1.114028.10.1.2 needs to be added to your list of certificates affected by this "Late CPS Update" incident.

Thanks for the feedback, the missing pre-certificates and certificates list have been uploaded. We still need to upload final certificates to CT.

Comment 6

[Ryan Dickson]

Hi Bruce,

One question about the timeline disclosed in the report and data we see in CT.

As I understand them, the EV Guidelines (Section 9.7 (3)) require all EV subscriber certificates to include the Issuer’s EV policy identifier (in this case 2.16.840.1.114028.10.1.2).

The timeline disclosed above indicates that for about 11 days, Entrust’s EV TLS CP OID was removed from TLS certificate issuance profiles. During that time, it looks like Entrust issued about 2,000 EV certificates without the Entrust EV TLS CP OID.

We also observe at least 4 EV certificates without the Entrust EV TLS CP OID issued after 9/22/23, which seems at odds with the disclosed timeline.

In any event, some of these certificates were recently revoked (e.g., https://crt.sh/?q=0fd0adac015b97aaae3ae8d97335de6c37bcfa155749a7f7434eee1b3196edb9 was revoked on 3/21/24), presumably necessitated by the absence of cPSURI (https://bugzilla.mozilla.org/show_bug.cgi?id=1883843).

Can you share:

• if you intend to open a separate incident report for the other issue described here (i.e., EV certificates issued without Entrust's EV TLS CP OID), like you did for https://bugzilla.mozilla.org/show_bug.cgi?id=1886467, since it appears to be a separate and distinct issue? If not, how do you recommend this is recorded? I'm asking because complete and transparent reporting creates opportunities for others to learn from the incident and avoid the same root cause, while also promoting continuous improvement.
• how this corpus of certificates missing Entrust’s EV TLS CP OID intersects (or doesn't) with those disclosed in https://bugzilla.mozilla.org/show_bug.cgi?id=1883843?

Thanks,
Ryan

Comment 7

[Bruce Morton]

(In reply to Ryan Dickson from comment #6)

Can you share:

• if you intend to open a separate incident report for the other issue described here (i.e., EV certificates issued without Entrust's EV TLS CP OID), like you did for https://bugzilla.mozilla.org/show_bug.cgi?id=1886467, since it appears to be a separate and distinct issue? If not, how do you recommend this is recorded? I'm asking because complete and transparent reporting creates opportunities for others to learn from the incident and avoid the same root cause, while also promoting continuous improvement.

Thanks for bringing this issue to our attention, we have submitted a sperate incident report regarding this issue, which can be found here: https://bugzilla.mozilla.org/show_bug.cgi?id=1888714

• how this corpus of certificates missing Entrust’s EV TLS CP OID intersects (or doesn't) with those disclosed in https://bugzilla.mozilla.org/show_bug.cgi?id=1883843?

You are correct, these certificates are test websites issued by “Entrust 4K TLS Certification Authority - EVTLS1” and “Entrust P384 TLS Certification Authority - EVTLS2”, which we will add to the list of impacted certificates. These certificates have all have been replaced or expired since.

Comment 8

[Ryan Dickson]

Hi Bruce,

Thank you for the response.

Given 1) the frequency we typically observe policy updates while processing cases in the CCADB and 2) the frequency of Entrust’s historical policy updates, we felt compelled to take a closer look at the updates described in the Timeline section of this report to better understand how they affected Entrust’s certificate issuance profiles in response to https://bugzilla.mozilla.org/show_bug.cgi?id=1888714, https://bugzilla.mozilla.org/show_bug.cgi?id=1883843, and https://bugzilla.mozilla.org/show_bug.cgi?id=1886467.

While doing so, we noticed what appears to be a lot of certificates issued that contain the Entrust EV TLS CP OID (2.16.840.1.114028.10.1.2), which was stated as not being used in EV SSL certificates at their time of issuance, violating the in-force CPS (Version 3.16, summarized below). Note: The start date for this query is intended to align with 3/18/2024, when we learned Entrust’s issuance profiles were updated to address https://bugzilla.mozilla.org/show_bug.cgi?id=1883843 (as disclosed in this comment). The timing is imprecise, as we don’t know exactly when changes were deployed or when updated policy documents were uploaded to Entrust’s policy repository. Consequently, the number of affected certificates may not be completely accurate.

Further examining the subjects of these certificates (example below), it seems many of these certificates may have been issued to replace the mis-issued certificates disclosed in https://bugzilla.mozilla.org/show_bug.cgi?id=1883843.

Example:

• This certificate was Issued on 11/15/23 and is missing the CPS URI, and now appears revoked.
• This certificate was Issued on 3/20/24 and contains the missing CPS URI (status is “good" as of this post).

Relevant Policy Language:

• What: Entrust CPS
• Version: 3.16
• Date: 2/20/2024
• URL: https://www.entrust.com/sites/default/files/documentation/licensingandagreements/entrust-certificate-services-cps-3-16.pdf
• Policy Commitments:

Certificates may include one or more of the following certificate policy identifiers:
- Document Signing Certificates: 2.16.840.1.114028.10.1.6
- Time-Stamp Certificates: 2.16.840.1.114028.10.3.5
- Verified Mark Certificates 2.16.840.1.114028.10.1.11

Effective 1 September 2023 the following certificate policy identifiers will not be used:
- S/MIME Certificates Class 1: 2.16.840.1.114028.10.1.4.1
- S/MIME Certificates Class 2: 2.16.840.1.114028.10.1.4.2

Effective 15 September 2023 the following certificate policy identifiers will not be used:
- SSL Certificates: 2.16.840.1.114028.10.1.5
- EV SSL Certificates: 2.16.840.1.114028.10.1.2
- Code Signing Certificates: 2.16.840.1.114028.10.1.3
- **EV Code Signing Certificates: 2.16.840.1.114028.10.1.2**

On March 21st, it appears a new version of the Entrust CPS was published that removed “EV SSL Certificates: 2.16.840.1.114028.10.1.2" from the “will not be used" list, and adds it to the “may" be used list.

Questions:

1. Can you help us understand the impact of these certificate policy changes and the certificates being issued during the described timeline?
2. Does Entrust consider these certificates to be mis-issued?

Thanks,
Ryan

Comment 9

[Ryan Dickson]

Following up, it seems as if the example I posted is included in https://bug1887753.bmoattachments.org/attachment.cgi?id=9393384.

I missed this entry due to relying on a less-specific crt.sh syntax (https://crt.sh/?q=77479548b34d9586011b2f7fc0c7c6be0a504a9915e1f5cb0049a4716f57a8f8 vs https://crt.sh/?sha256=77479548B34D9586011B2F7FC0C7C6BE0A504A9915E1F5CB0049A4716F57A8F8).

However, https://crt.sh/?sha256=854DE48308E07F61557EDE81B672CB98B9D3D902BB1DB6EF3E30331B03C1E3DF is an example of another (pre-)certificate that I cannot find on any of the lists attached to this report.

Comment 10

[Bruce Morton]

(In reply to Bruce Morton from comment #5)

(In reply to Rob Stradling from comment #2)
Rob, the certificate lists were incomplete due to an export problem between SQL and Excel. The SANS field exceeded SSMS’ 8k field limit when exporting, which threw off the field formatting when importing into Excel. Below are he certificates which were missed:

Pre-certificates:
https://crt.sh/?sha256=854DE48308E07F61557EDE81B672CB98B9D3D902BB1DB6EF3E30331B03C1E3DF
https://crt.sh/?sha256=504AFECFE1380B3D9B1B4337AD1FC6132A0EB9B9FFD919159DA365ABAF91AE50
https://crt.sh/?sha256=9E72ED84D1E0F0CC8E0FAA9D17B8BE97998CC89D4141BA38ABDAE0C329305A3F
https://crt.sh/?sha256=34083619687C0AFE3C223056A5C2552532EDD6713260FD42F4482B06E2B20F53
https://crt.sh/?sha256=2D6F0088F3D33F9265ACB7B916D0BDF346C09D58526F9465BA7374DA681FC90E
https://crt.sh/?sha256=70BCA1689E202722478CEDB83A28AC8565375F147AF0CABD1E44F61782B3DAFD
https://crt.sh/?sha256=AB322BEB18BA6F35BE38238EAD7520ACD55679EDE1DE22862D9F95E72EB54ADE
https://crt.sh/?sha256=A4177958362286663BF8355D83E06A076B33AE9AD1E8DB06372853DB550C17E4
https://crt.sh/?sha256=A6B9B8CAA1EEEA1EE16600EB38902620176B16D285CCACB35472CB2157BC3BAC

Leaf certificates:
https://crt.sh/?sha256=B5890152A71FA37DED29BDC99706DF8D56C9FC044C25CF8D698B5A72ABA2540B
https://crt.sh/?sha256=DD0E61F65F886E4892521EE1EA7948B53A10FAA00E369AD8743744C437EB5430
https://crt.sh/?sha256=EBB72130F47A2137557FFB58A26DD2CBE1B0779F676BE8F6CE14B37F8D6FE35F
https://crt.sh/?sha256=C843A434746F2E8E503BE95321E3E328AF5BB78EE7A1EE9865FE15227873FC0A
https://crt.sh/?sha256=21607C6FBB332E237229500307EA1980C8C23042A55601284BD30813B1AF18CF
https://crt.sh/?sha256=EFA994A4AEBDD77636432E8304FFB70F343B1868EF3FB468662CA8AA449A98E4
https://crt.sh/?sha256=9857AF9086591DECA3D185399F7449289690CA921F9206E73BBB9AB8C38931B6
https://crt.sh/?sha256=78F4C933E45A30AE2029D63BABE82636CCA9436C83FFA1389C1A8184D2D188A1
https://crt.sh/?sha256=493D1EE5991D94F4F7F10B0FEDE665C517223C43F9CA28A9F828AF059D3EE678

Comment 11

[Bruce Morton]

(In reply to Ryan Dickson from comment #8)

Questions:

1. Can you help us understand the impact of these certificate policy changes and the certificates being issued during the described timeline?

We believe that there was no impact to the ecosystem because the affected certificates included the CA/Browser Forum reserved Certificate Policy identifier. However, some browsers did not identify the affected certificates as EV certificates because the Entrust Certificate Policy Identifier was not included during this time.

2. Does Entrust consider these certificates to be mis-issued?

The certificates are considered mis-issued because they were issued with the Entrust Certificate Policy identifier, which was not included in the CPS.

The 33,754 certificates in this incident can be divided into two categories:

A: 24,709 certificates issued between September 11, 2023 17:46:00 UTC <> March 18, 2024 21:33:53 UTC
These certificates are being replaced and reissued as result of the incident reported in bug https://bugzilla.mozilla.org/show_bug.cgi?id=1883843.

B: 9,045 certificates issued between March 18, 2024 21:33:53 UTC <> March 21, 2024 16:34:44 UTC

The certificates were issued as a replacement of the incident reported in bug #1883843. These certificates were issued in accordance with the EV Guidelines and with the certificate profile intended by Entrust.

If the certificates were re-issued, they would be re-issued with the identical certificate profile. As such, we are not planning to revoke.

We will file an incident report detailing the exceptional circumstances that led to our decision not to revoke the affected certificates in category B.

Comment 12

[Bruce Morton]

Attachment: 20240408-EV-TLS-Certificates-OID-CPS-finalcerts-all.txt

This is an update of all certificates for this incident.

Comment 13

[Bruce Morton]

Posted Failure to revoke EV TLS certificates issued before CPS update, https://bugzilla.mozilla.org/show_bug.cgi?id=1890685

Comment 14

[Bruce Morton]

Action item to "Review and update release and emergency release process to ensure compliance documentation stays synchronized" has been moved to 30 June 2024.

Comment 15

[Bruce Morton]

Updating action item table per comment 14.

Action Item	Kind	Due Date
Review and update release and emergency release process to ensure compliance documentation stays synchronized	Prevent	2024-06-30

Continue monitoring bug for comments.

Comment 16

[Bruce Morton]

We have no updates for this week and will continue to monitor the bug.

Comment 17

[Bruce Morton]

We have no updates for this week and will continue to monitor the bug.

Comment 18

[Ben Wilson]

Setting next update to 2024-06-17 to coincide with status reports set in Bugs #1890896, #1883843, and #1890123.

Comment 19

[ngook.kong]

We have no updates for this week but will have an update on "Review and update release and emergency release process to ensure compliance documentation stays synchronized" by 6/30 per our posted due date in Comment #15
https://bugzilla.mozilla.org/show_bug.cgi?id=1887753#c15

We will continue to monitor the bug.

Comment 20

[Bruce Morton]

We have no updates, but plan to update action status this week.

Comment 21

[Paul van Brouwershaven]

All action items in this bug have been completed, we request this bug to be closed.

Action Item	Kind	Due Date
Review and update release and emergency release process to ensure compliance documentation stays synchronized	Prevent	Complete

Comment 22

[Paul van Brouwershaven]

We have no updates and all action items have been completed, we request this bug to be closed.

Comment 23

[Ben Wilson]

I'll close this on or about Friday, 12-July-2024. However, please note that we may add action items related to this bug to Bug #1901270 for Entrust to address. Such items would be required as remediation for compliance issues identified in the June 21 report and in any responses from Mozilla and the Mozilla community.