1516561 - DigiCert: Underscores - Canadian Imperial Bank of Commerce

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

All essential notice dates:

1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
3.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
4.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
5.	October 26, 2018 – Final ballot was proposed. 
6.	November 2, 2018 – Voting period starts
7.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
8.	November 19, 2018 – We first hear of customers not being able to meet the revocation timeline.
9.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
10.	January 26 2018 – Proposal on when all certs will be revoked.

Customer was given a list of all their impacted certificates on Dec 4, 2018. Blackout period is from December 15th to January 1st. 

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 

1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 1, 2018 – We cease issuance of underscore characters in case the discussion goes south (obviously it does) 
3.	October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Bad data leads to only some customers being notified. 
4.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
5.	October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article
6.	October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems.
7.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
8.	October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire.
9.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
10.	October 19, 2018 – Internal discussion to start comms about CAB Forum plan.
11.	October 20, 2018 – Second emergency meeting to start comms process.
12.	October 24, 2018 – Gather of data on all impacted certs across the different systems
13.	October 26, 2018 – Final ballot was proposed. 
14.	November 1, 2018 – We notice the data is wrong and regather the information.  
15.	November 2, 2018 – Voting period starts
16.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
20.	November 29, 2018 – Posting to Mozilla about concerns with ballot
21.	November 29, 2018 – Final comms is dropped about the ballot and its impact. 
22.	November 30, 2018 – Final internal advisory on issue.
23.	November 20, 2018 – Customer given list of certificates and advised to participate in the Mozilla discussion. All exceptions to the revocation date are denied. People to start to escalate to demand that there is an exception process, we just don't know about it yet.  
24.	December 7, 2018 – Customers engage with Mozilla community
25.	December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline
26.	December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore.
27.	December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs.  The goal is to provide better information on the scope of impact.
28.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
29.	January 26, 2018 - Proposal on when all certs will be revoked.
 
3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements.  All certs for this particular entity will be revoked on January 26.
 
4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 

These particular systems follow a strict change management process that requires verification in a lower environment before getting approval to implement in production. These particular certs do not require public trust in Mozilla or Chrome bu were already used on the resources when the CAB Forum made the change requiring revocation. Note these are on the legacy Symantec hierarchy, meaning testing is required with the DigiCert root. The certificates are already NOT trusted in Chrome or Mozilla. 

5.	The complete certificate data for the problematic certificates. 
Listed below. 

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Replacing the certificates and changing the domain name require identical efforts. Instead of making two changes to the system, a single change to update both domain and certificate at the same time is easier.

7.	 List of steps CA is taking to resolve the situation and ensure it will not be repeated. 
Working on it. We've got some automated install tools that should be available soon. we're also contemplating how to effectively separate out the MS and Apple ecosystems from Mozilla and Google.




Actual results:

Certs:
5217b2cbf0288db1a0a8e0d949d1503a	https://crt.sh/?id=78360862
0335c097f2cef2a521aca27a89030f82	https://crt.sh/?id=142988350
48f85acac32c6577232a7c155f9096db	https://crt.sh/?id=146398248
6f2f14386f7d2176b2e2c9fe27734958	https://crt.sh/?id=178188394
55ec5680d6ee7a716da0c7eba455de11	https://crt.sh/?id=215207765
293b6a4fa1dbbca10bd3a6867dd475ae	https://crt.sh/?id=215207764
06927fa78ff8db42f4606182ba8d835f	https://crt.sh/?id=251431872
23c9bd52c62093c4e1b8fe66c8de667c	https://crt.sh/?id=251431875
6a699a47c6fe31b82f6b8c32266481a6	https://crt.sh/?id=251431904
3b0e64b7104a03d3f5115bd2aa441add	https://crt.sh/?id=251431900
40e6bc1ba53829570f1f9f8ae6ae5387	https://crt.sh/?id=251431897
4831810f818586216366f1237871c63f	https://crt.sh/?id=251431894
3efd3791417fb9ada4b16767886f7fcc	https://crt.sh/?id=251431891
5b2d0d63d1692e7f37f7de6a267100f5	https://crt.sh/?id=251431888
7614325ed1ad28c4c0aaa984ba82394d	https://crt.sh/?id=251431885
43e605d1512305faec99ec2165183fcb	https://crt.sh/?id=251431882

Comment 1

[Brenda Bernal]

Additional input from the customer below:

"We will be able to replace the certificates. The issue is that we are not given enough time to do it. For one application the underscore will be removed and the certificate will be issued under the Digicert root. For the other impacted application, we will keep the underscore but migrate the certificate to our internal CA. In both case, modification to the applications and infrastructures are required and consequently have more risks than a usual certificate renewal. Because of that we must test the change in lower environment and follow a strict change management process that requires validation in lower environment before getting approval to implement in production.

For that reason, we would appreciate if the certificates with underscore were not revoked until February 28, 2019."

Comment 2

[Ryan Sleevi]

The original report said:

29. January 26, 2018 - Proposal on when all certs will be revoked.

And the modified now says

For that reason, we would appreciate if the certificates with underscore were not revoked until February 28, 2019.

I'm curious what lead to the change in evaluation of timelines and risk?

Comment 3

[Brenda Bernal]

Hi Ryan, The customer was aggressively targeting to get all certs replaced by this January month end. They asked for a sufficient period of time to ensure they can address any issues during the change process if a rollback is required. They want to ensure a successful implementation before the February 28, 2019 date.

Comment 4

[Ryan Sleevi]

Brenda: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?

Comment 5

[Brenda Bernal]

I can confirm that an incident has occurred and the details provided are accurate to the best of our knowledge. Our planned extension to revoke the remaining certificates (listed above) is 28-February-2019. We will provide periodic updates as progress is made.

Comment 6

[Brenda Bernal]

Update: All remaining underscore certs, as noted above, have either expired or were revoked.