1516599 - DigiCert: Underscores - Ericsson

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
3.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
4.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
5.	October 26, 2018 – Final ballot was proposed. 
6.	November 2, 2018 – Voting period starts
7.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
8.	November 19, 2018 – We first hear of customers not being able to meet the revocation timeline.
9.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
10.	April 30, 2018 – Proposal on when all certs will be revoked.

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 

1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 1, 2018 – We cease issuance of underscore characters in case the discussion goes south (obviously it does) 
3.	October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Bad data leads to only some customers being notified. 
4.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
5.	October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article
6.	October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems.
7.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
8.	October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire.
9.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
10.	October 19, 2018 – Internal discussion to start comms about CAB Forum plan.
11.	October 20, 2018 – Second emergency meeting to start comms process.
12.	October 24, 2018 – Gather of data on all impacted certs across the different systems
13.	October 26, 2018 – Final ballot was proposed. 
14.	November 1, 2018 – We notice the data is wrong and regather the information.  
15.	November 2, 2018 – Voting period starts
16.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
20.	November 29, 2018 – Posting to Mozilla about concerns with ballot
21.	November 28, 2018 – Final comms is dropped about the ballot and its impact. 
22.	November 30, 2018 – Final internal advisory on issue.
24.	December 7, 2018 – Customers engage with Mozilla community
25.	December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline
26.	December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore.
27.	December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs.  The goal is to provide better information on the scope of impact.
28.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
29.	April 30, 2018 – Proposal on when all certs will be revoked.
 
3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements. 

However, 30 day certificates will not work in this case because it will lead to double work . The certificates are deployed to customer installations, which require downtime and change windows. New FQDNs need to be rolled out and impact additional applications so better to do it at once. 
 
4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 

72 certs are being replaced by wildcard certs. The remaining certificates all need to be individually dealt with through deployed customer installations. The obstacle is buy-off from third party (two removed from DigiCert). 11 certs will expire, which leaves 74 that need to be replaced individually. 

5.	The complete certificate data for the problematic certificates. 
Listed below. 

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The freeze period is Dec 16-Jan 13. They are planning the replacement and will start immediately after the freeze is lifted. Negotiating downtime and change windows is the delay.

7.	 List of steps CA is taking to resolve the situation and ensure it will not be repeated. 
Still getting this info. They are aware of the need for a better process. Will post an update when we work something up.


Actual results:

Certs:
3977ebab52270d562b183a31cf07177a	https://crt.sh/?id=22338181
11d6447549aab5048028125276931db4	https://crt.sh/?id=22338417
646f0381d41eb777e48c6c7191ce3da4	https://crt.sh/?id=36275795
0a2489c21b3de1f1011781bcc2dde500	https://crt.sh/?id=43299779
3cbdbdff3aa2f92d8c70dbe47cacd21a	https://crt.sh/?id=43299837
74289fd0ef5aeab0aaa47baa9dc9a8ec	https://crt.sh/?id=48168620
442d6d7f077c0aa28fda609c3b9b3c63	https://crt.sh/?id=54337593
444f5ed6bfe2571e25b479da1eadacb7	https://crt.sh/?id=59628913
10033f8c0e00076f2b639e32e83162d1	https://crt.sh/?id=60284631
55c0f75e759962a4ea493f7629f8260a	https://crt.sh/?id=62536568
5a3cca56ad55edd9d0f6f081a76aaf94	https://crt.sh/?id=62536655
673fb4a830bf5c2d541ee14f23108f6e	https://crt.sh/?id=159974422
0c6436b5f1561ecc61f676606fc5b138	https://crt.sh/?id=159974448
47de33e25a3a76e6c86a297994d8fda2	https://crt.sh/?id=161918383
7b7ad30889fcc205af44fe5cb4e3fe1e	https://crt.sh/?id=161918400
1f7dd110c03bf97a02356daa3f961108	https://crt.sh/?id=161918411
41027867ef42106d823ad068a04be761	https://crt.sh/?id=161918397
47b08a101c270bda3b7a4a019fb148da	https://crt.sh/?id=161918399
669e48d3a81b6d7a06f784facbf1a16b	https://crt.sh/?id=161918403
12a4569951d8a375ce346ac731e755d5	https://crt.sh/?id=171387330
25edc88e7a4750d1bc0a8f2294513d19	https://crt.sh/?id=176217303
2f2464e577d755aff46c83386f4e5e76	https://crt.sh/?id=176217371
21bc36f43fe26b7a3949c5904fe207f2	https://crt.sh/?id=197644173
384a843a063189dfe10bd37166851d33	https://crt.sh/?id=197644055
59b0a9fd8031d1c833c8b1f7a4f66594	https://crt.sh/?id=205758807
793804e585e7c5d4cd77327a1c7ca5ec	https://crt.sh/?id=205758827
6876d2c366698bcdd4af9f44cdee5349	https://crt.sh/?id=206192605
7e220641d2eaff0f71e435ab96a30e9c	https://crt.sh/?id=206192668
27bd147686e1729cd85a79b5a2f8b6e4	https://crt.sh/?id=234189060
2b953afc44dc11d201d45b3b0cbfcf34	https://crt.sh/?id=234189068
4be9d8436a8cb0174f4c84adffd5b449	https://crt.sh/?id=234189072
5c18f40ac307b7a05b56bafa312e6e8d	https://crt.sh/?id=239151880
39d52994416898f31bba5d5b4037298c	https://crt.sh/?id=239151883
8d3f35dddcc4224aa48214e4cc771a	https://crt.sh/?id=239151890
7ffbec9b803c29f6a5e63648ff105c17	https://crt.sh/?id=239151895
532c8f3931780f4c6bb36149b2c1eae1	https://crt.sh/?id=239151912
51cf1d226ac7a1093db173c305d49dc0	https://crt.sh/?id=239151921
4d75bfb99ece05c158796e99082e73a0	https://crt.sh/?id=249659019
34a88f423b8d05ced34e2b1fe5b58d23	https://crt.sh/?id=249659022
09951a2fbb3f3619f6f645dbe5c0085a	https://crt.sh/?id=249659026
209bde1eead87762de25afb13d96ec89	https://crt.sh/?id=249659032
3ab7605fa8846406da486654f0139f18	https://crt.sh/?id=254724838
0afdc635cb7c3967755fbb849f3d9126	https://crt.sh/?id=254724914
3d518df386000d52c60079c80018e256	https://crt.sh/?id=254724920
69f97902d9467208817b9a05a88a7802	https://crt.sh/?id=254724861
4c69d21c89e3b2ea38742456ffbd2af4	https://crt.sh/?id=254743440
149748e7dde901eace17be5f54f8d91d	https://crt.sh/?id=254743460
23be29eab449b66c5d8aac53c2f4310c	https://crt.sh/?id=254743591
69fc165ef71b4cc34a72c6dab0e71f3e	https://crt.sh/?id=254744440
529626e82f8cd5ea6fb78c28a4ee085f	https://crt.sh/?id=255542880
409820cb1726928651bf21e241a45661	https://crt.sh/?id=255542882
54f2c3ccb2966bf0d0db7afd0d308b77	https://crt.sh/?id=255542885
26afea22f069350640d68db7e9453af7	https://crt.sh/?id=255542884
730969b39689ac5144e1ae647fe9c11b	https://crt.sh/?id=255542887
6b838384f940a5fff44eeb11a665e0bc	https://crt.sh/?id=255542890
6306b0597ad65adc67209a7a062ceaf6	https://crt.sh/?id=255542894
6ebcec2e85096c7a0d5480d81c085cfb	https://crt.sh/?id=255542900
2b55ccd4abd9c41f4094372cbe2ca0a7	https://crt.sh/?id=255542896
7abc80cf9ecfd7646e6305dd1780738e	https://crt.sh/?id=255542901
7059e769ee17ea44d47e16d92fb8b5dd	https://crt.sh/?id=255542904
6efb8152bf5fb1dab90515308b6593f9	https://crt.sh/?id=255542906
3b2fca615af831161639674f8382ec25	https://crt.sh/?id=255542908
055c96d876e0b63b44c3ca5b9db462f6	https://crt.sh/?id=271004515
04178b78a189712a3009d05d5e6b5aad	https://crt.sh/?id=271007351
0f9580aa61a5d3028c9a651361e5011f	https://crt.sh/?id=273545463
0b4dd0b423c1db8bb41aaf1282508fa6	https://crt.sh/?id=273557968
073f99a78ac37d4df8714640147c3680	https://crt.sh/?id=273569089
0262767f6a5fbc40be6d99b7f7afeaeb	https://crt.sh/?id=311662455
01ec5194252368cf43b56319b15e6ecd	https://crt.sh/?id=311662049
05edf706a5ecd2e153f232a646650ac1	https://crt.sh/?id=315857566
07eb34d2edb41ad197356db15e600bf1	https://crt.sh/?id=355192405
020868cc760a04cf68709f785336a56d	https://crt.sh/?id=355195641
08bc6e9a3750f57926f4642298faf970	https://crt.sh/?id=367720136
0c6ddae47470cfcb873de522e4b29f48	https://crt.sh/?id=539442119
07c1eca6edf9e9af14511013e3b29788	https://crt.sh/?id=543390138
037a46710cb0a5abcf445a09e54a8391	https://crt.sh/?id=555463042
0d5bd8851788c68069708de46311279d	https://crt.sh/?id=584063508
0a561ad65fd0daf2ea24778f0ae8cd81	https://crt.sh/?id=627186062
062a653cb27e9560d73878849da84ca7	https://crt.sh/?id=627187516
0bba6bc08693c75ae661fc735a2fa86f	https://crt.sh/?id=628071865
0186aaeffa50106c761aaa235e8f95fb	https://crt.sh/?id=628072553
0f68d6bf4cd152c390bfec9fe769c834	https://crt.sh/?id=628086241
03c58b5caade42739f9b87f67506f006	https://crt.sh/?id=628086132
091342de436286e1c65a7824faad0c26	https://crt.sh/?id=628086326
04d4b631028328a9cecf1d2b8230ef2c	https://crt.sh/?id=628086509
07800dcc9f245179dacbcfb9bdb00983	https://crt.sh/?id=628086844
0455fbeb4b4364e21a8fcb09c7ed5d42	https://crt.sh/?id=628087032
07bb11db7ec26ba4bc5bf31a605730fd	https://crt.sh/?id=628087078
074312ca99ef3faaa061da1ea13132a8	https://crt.sh/?id=628089101
0c4a2ba30f587f96267989d45d1cc3cc	https://crt.sh/?id=628091401
0b88ffba6d07971880665180bb2278f5	https://crt.sh/?id=628095660
07385b1cc306aee82d04c90976b453e7	https://crt.sh/?id=628095684
023dc21ba758d2bc8e1d9e5a9e33d531	https://crt.sh/?id=628095714
0e81a9a29d537ef67a6bdcb8362ae277	https://crt.sh/?id=628095754
052afb4ceb71a17f31f03f95fbdb35d8	https://crt.sh/?id=628097298
0414d8632430e5daceecf6cd13a067cd	https://crt.sh/?id=628098602
0e7e24c933a4825b70b879cf0a724b86	https://crt.sh/?id=628103967
07a6c1aa421b296d3bdee86987e132fd	https://crt.sh/?id=628104117
042f76f5dec774af9e8a04d32d87c0d7	https://crt.sh/?id=628103938
06274053a69fdb145f6b47c02350eba0	https://crt.sh/?id=628103989
0f1115bcddf43e9eb5e9f4e9807ae03b	https://crt.sh/?id=628104037
02b98cc2d2a0b81b80885e1ad311a47d	https://crt.sh/?id=628104059
05658962ffd26e904a8b5a5cdc7dcbb3	https://crt.sh/?id=628104086
054269143ecbf88d0e09449b8a9e2a32	https://crt.sh/?id=628104136
04a4d134fc10bc9a4495685c3f7f3c12	https://crt.sh/?id=628104186
0f0341483b8748d34cfa95fac4ed10de	https://crt.sh/?id=628104205
04d420aad204c248484f89b4573d1006	https://crt.sh/?id=628109469
05a8c697ede7d05db50f7d16b21c0416	https://crt.sh/?id=628109533
0efb50333f937b9224e7b7552bd36463	https://crt.sh/?id=628109552
0f55dd2d9a982fe7051e2ea9cb885d13	https://crt.sh/?id=628109597
0ea8478ba02ead773aaf68f52c904647	https://crt.sh/?id=628109615
03ab6ef2d619af80da784107f5e4a0b4	https://crt.sh/?id=628109630
08c980d703b7df5316aa102b25bcf698	https://crt.sh/?id=628109660
01f832e4d82053c79cbe4590fbf7296a	https://crt.sh/?id=628109500
0f90e79864750755fe13c9fab34cb20f	https://crt.sh/?id=628109771
0d0e8693db60baec58bede4a59c69f2c	https://crt.sh/?id=628109840
0ae13d10861e9d874e06ed6b63b03d0e	https://crt.sh/?id=628118332
063b4000cc3d12c2516b5e8170b7d26b	https://crt.sh/?id=628118695
033390d7cf24cd0fb34083de601043a9	https://crt.sh/?id=628118288
0ff09413f4d4ec0dde4736006eee5248	https://crt.sh/?id=628118614
09eeb31b911c310f1517befbdf66a075	https://crt.sh/?id=628122694
0aa1998113af273035845e110de26571	https://crt.sh/?id=628126109
09de560f8f93a830b2a37a4475cb76e1	https://crt.sh/?id=628126137
0bcd4dc9ae103ba8049c9b2311eb575b	https://crt.sh/?id=628126251
01ea5363afce0292ef8b5d324dd74b16	https://crt.sh/?id=628128395
06f3dc9a78707f1ef624df52085c823d	https://crt.sh/?id=628135161
0b5ded168dd58e5e5b45ac1851d0d321	https://crt.sh/?id=628135354
09e188f89a3677a71c6842335d18e042	https://crt.sh/?id=628135522
03eccf9e4f0a1ba212f5893a4b90dcf3	https://crt.sh/?id=628148634
0d089bd5a15e71224b3af53801685388	https://crt.sh/?id=628154756
05f5a3d3b40beac62c7471ba0635a143	https://crt.sh/?id=628154962
09fb0626b789597442763acd2cff332a	https://crt.sh/?id=628159694
0e10aa723c15ac54a01aedc84f7455cc	https://crt.sh/?id=628160193
09308fa0f31f4e849b4df8dd60582cd5	https://crt.sh/?id=628160424
08c31056a9c75b570bfd74a5c96c25d9	https://crt.sh/?id=628160453
03d7a937080e1f2402f9ab31976eb8b4	https://crt.sh/?id=636167641
0b226d725215de9ef65d12d92d16a7d5	https://crt.sh/?id=664103699
0c6137aab3dc738e8c7cdd2397955596	https://crt.sh/?id=664104899
085869391c7bee642b16adba8a7ededf	https://crt.sh/?id=664105913
0e9236ce0bed6bd9dc591a8b41e49101	https://crt.sh/?id=664115209
03195434695c63f728d644f700754219	https://crt.sh/?id=664115140
0d9eef8fd428760b50d51d282fe70fae	https://crt.sh/?id=781591132
0f7078145386905a1cd9fc08f1d32f55	https://crt.sh/?id=781591239

Comment 1

[Ryan Sleevi]

(In reply to Jeremy Rowley from comment #0)
> 6.	Explanation about how and why the mistakes were made or bugs introduced,
> and how they avoided detection until now.
> 
> The freeze period is Dec 16-Jan 13. They are planning the replacement and
> will start immediately after the freeze is lifted. Negotiating downtime and
> change windows is the delay.
> 
> 7.	 List of steps CA is taking to resolve the situation and ensure it will
> not be repeated. 
> Still getting this info. They are aware of the need for a better process.
> Will post an update when we work something up.

I think this remains an important thing to understand, and I'm hoping you have an update here as the deadline approaches.

Understanding what steps are or will be taken to ensure that future areas of non-compliance are not revoked in a timely fashion due to "negotiating downtime and change windows" is an important step in ensuring there is meaningful progress.

For example, have negotiations begun? How long are they expected to take? Is there a reason the downtime and change window cannot be performed on Jan 14? 

> 29.	April 30, 2018 – Proposal on when all certs will be revoked.

There is not sufficient detail, at present, to understand how that conclusion was reached and what information was used to reach it. I'm hoping that for the remaining 74 certificates, DigiCert can share more data about how it (potentially) reached the decision that it was necessary to violate the BRs. Based on this bug alone, it merely appears that "They told us that's when they could do it", which is not particularly inspiring in-and-of-itself.

Comment 2

[Jeremy Rowley]

Based on the conversation on the forum, the post from Wayne, and instruction from Google, our understanding is there is no exception or extension possible and the expectation is that all CAs will revoke the certificates on the date required by the BRs. We hope that the same rules/penalties/expectations will be applied to those CAs who fail to revoke on the required date. Thank you for the discussion. Although we were hoping for more compassionate results, we do appreciate the feedback and clarification on expectations.

Comment 3

[Jeremy Rowley]

Seems there was a mis-communication on the intent of the discussions. We will post an update answering Ryan's questions tomorrow. Please ignore my previous post. Apologies for the confusion.

Comment 4

[Ryan Sleevi]

Jeremy: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?

Comment 5

[Brenda Bernal]

Update: All remaining underscore certs for this customer has been revoked as of today (30-April-2019).