1515564 - DigiCert: Underscore character certificates

Status: RESOLVED DUPLICATE of bug 1515788

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Hard question to answer for a future bug. However, for completeness and the people who don’t follow the CAB Forum here’s the timeline, I figured I'd include all essential notice dates:

1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
3.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
4.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
5.	October 26, 2018 – Final ballot was proposed. 
6.	November 2, 2018 – Voting period starts
7.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
8.	November 19, 2018 – We first hear of customers not being able to meet the revocation timeline.
9.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
10.	April 30, 2018 – Proposal on when all certs will be revoked.

If you're talking about prior to this ballot, we were unaware that underscore characters were not allowed. With ballot 202 failing, we weren't sure where that left the industry, especially consider 1034's age, applicability, and what we thought was a goal to secure all websites. Apologies for being incorrect on my reading of the requirements here. I did propose 202 originally to try and clear up the confusion. 

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 
Not sure the best method of presenting this info, but you’ll likely get the most accurate picture with it layered in with the timelines above. I left off the discussion timelines in the CAB forum as that’s all a matter of public record and there is a lot of discussion. Let me know if you want me to add that to this record.
1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 1, 2018 – We cease issuance of underscore characters in case the discussion goes south (obviously it does) 
3.	October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Bad data leads to only some customers being notified. 
4.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
5.	October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article
6.	October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems.
7.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
8.	October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire.
9.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
10.	October 19, 2018 – Internal discussion to start comms about CAB Forum plan.
11.	October 20, 2018 – Second emergency meeting to start comms process.
12.	October 24, 2018 – Gather of data on all impacted certs across the different systems
13.	October 26, 2018 – Final ballot was proposed. 
14.	November 1, 2018 – We notice the data is wrong and regather the information.  
15.	November 2, 2018 – Voting period starts
16.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
17.	November 9, 2018 – Outreach to customers is complete. 
18.	November 19, 2018 – We first hear of customers not being able to meet the revocation timeline.
19.	November 20, 2018 – We discover more data troubles and that the data sent from our BI team is incomplete. 
20.	November 29, 2018 – Posting to Mozilla about concerns with ballot
21.	November 29, 2018 – Final comms is dropped about the ballot and its impact. 
22.	November 30, 2018 – Final internal advisory on issue.
23.	December 4, 2018 – We advise customers who have concerns to participate in the Mozilla discussion and tell customers there is no exception process.
24.	December 7, 2018 – Customers engage with Mozilla community
25.	December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline
26.	December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore.
27.	December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs.  The goal is to provide better information on the scope of impact.
28.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
29.	April 30, 2018 – Proposal on when all certs will be revoked.
 
If you're talking about post-202, I didn't do anything to stop issuance. The ballot failed to I thought that left us in another weird gray state for compliance. My plan was to propose another version of ballot 202. Unfortunately, the governance discussions took over, and I never got around to proposing something with changes to clarify the situation. When we acquired Symantec, we found they had the same confusion we did about the status of underscore characters. The certs are a mix of legacy Symantec and legacy DigiCert.

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements. 
 
4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 

The problematic certificates are all the same. They include underscore characters and couldn’t be revoked by Jan 15, 2018 because of corporate blackout periods at the subscriber end. The current plan is to complete all revocation by April 30, assuming the browsers do not identify any overt concerns with the missed deadline.  I’m sure we’ll be revoking certificates throughout the period.

5.	The complete certificate data for the problematic certificates. 
Attached. All are CT logged now. Some don't have a crt.sh link yet because we logged them tonight. 

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

I’m not sure how to answer this question. The issue is with the revocation timeline and with the date (Jan 15th). Companies have blackout periods from Halloween until mid-Jan or mid-Feb, depending on the company. They can’t replace a certificate during that time or change their domain space. 

One thing we could have done better is communicate the change. The biggest challenge there was that the data is still split across the various systems. We’ve been working towards consolidation of the Symantec and DigiCert platforms but haven’t deprecated the legacy Symantec systems yet. We are migrating customers from the legacy Symantec system to CertCentral (the DC platform), but that projected work won’t complete until end of 2019 or mid-2020. We mostly consolidated backends so we have most of the data. However, people still using Symantec certificates were difficult to discover. Note that the number of people using legacy Symantec certs is why we wondered if they were in scope.  Legacy Symantec systems prove…difficult to get data from.

If you're talking about underscore certs in general, then I probably should have updated and pushed 202 again right after the previous one failed.

7.	 List of steps CA is taking to resolve the situation and ensure it will not be repeated. 
a.	We’ve reached out to every customer and engaged in communication with them about the obstacles in revoking the certificates
b.	We’ve identified that anytime between Oct 31-Feb 15 is a really bad time to revoke certs when we can’t describe a security issue related to the certificates. We'll make sure to raise that concern as it occurs. 
c.   As mentioned, we need better data. This is coming as we collapse legacy Symantec systems.

Comment 1

[Jeremy Rowley]

Attachment: underscore.xlsx

Comment 2

[Wayne Thayer]

Discussion is at https://groups.google.com/d/msg/mozilla.dev.security.policy/0oy4uTEVnus/pnywuWbmBwAJ

Comment 3

[Wayne Thayer]

Closing this because it appears that DigiCert will be creating separate bugs for each affected customer, e.g. 1515788.