DigiCert: Underscores - Ericsson
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: jeremy.rowley, Assigned: brenda.bernal, NeedInfo)
Details
(Whiteboard: [ca-compliance])
User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0 Steps to reproduce: 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. 1. September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 2. October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance 3. October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal 4. October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 5. October 26, 2018 – Final ballot was proposed. 6. November 2, 2018 – Voting period starts 7. November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs. 8. November 19, 2018 – We first hear of customers not being able to meet the revocation timeline. 9. January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course) 10. April 30, 2018 – Proposal on when all certs will be revoked. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 1. September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 2. October 1, 2018 – We cease issuance of underscore characters in case the discussion goes south (obviously it does) 3. October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Bad data leads to only some customers being notified. 4. October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance 5. October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article 6. October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems. 7. October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal 8. October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire. 9. October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 10. October 19, 2018 – Internal discussion to start comms about CAB Forum plan. 11. October 20, 2018 – Second emergency meeting to start comms process. 12. October 24, 2018 – Gather of data on all impacted certs across the different systems 13. October 26, 2018 – Final ballot was proposed. 14. November 1, 2018 – We notice the data is wrong and regather the information. 15. November 2, 2018 – Voting period starts 16. November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs. 20. November 29, 2018 – Posting to Mozilla about concerns with ballot 21. November 28, 2018 – Final comms is dropped about the ballot and its impact. 22. November 30, 2018 – Final internal advisory on issue. 24. December 7, 2018 – Customers engage with Mozilla community 25. December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline 26. December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore. 27. December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs. The goal is to provide better information on the scope of impact. 28. January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course) 29. April 30, 2018 – Proposal on when all certs will be revoked. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements. However, 30 day certificates will not work in this case because it will lead to double work . The certificates are deployed to customer installations, which require downtime and change windows. New FQDNs need to be rolled out and impact additional applications so better to do it at once. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 72 certs are being replaced by wildcard certs. The remaining certificates all need to be individually dealt with through deployed customer installations. The obstacle is buy-off from third party (two removed from DigiCert). 11 certs will expire, which leaves 74 that need to be replaced individually. 5. The complete certificate data for the problematic certificates. Listed below. 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. The freeze period is Dec 16-Jan 13. They are planning the replacement and will start immediately after the freeze is lifted. Negotiating downtime and change windows is the delay. 7. List of steps CA is taking to resolve the situation and ensure it will not be repeated. Still getting this info. They are aware of the need for a better process. Will post an update when we work something up. Actual results: Certs: 3977ebab52270d562b183a31cf07177a https://crt.sh/?id=22338181 11d6447549aab5048028125276931db4 https://crt.sh/?id=22338417 646f0381d41eb777e48c6c7191ce3da4 https://crt.sh/?id=36275795 0a2489c21b3de1f1011781bcc2dde500 https://crt.sh/?id=43299779 3cbdbdff3aa2f92d8c70dbe47cacd21a https://crt.sh/?id=43299837 74289fd0ef5aeab0aaa47baa9dc9a8ec https://crt.sh/?id=48168620 442d6d7f077c0aa28fda609c3b9b3c63 https://crt.sh/?id=54337593 444f5ed6bfe2571e25b479da1eadacb7 https://crt.sh/?id=59628913 10033f8c0e00076f2b639e32e83162d1 https://crt.sh/?id=60284631 55c0f75e759962a4ea493f7629f8260a https://crt.sh/?id=62536568 5a3cca56ad55edd9d0f6f081a76aaf94 https://crt.sh/?id=62536655 673fb4a830bf5c2d541ee14f23108f6e https://crt.sh/?id=159974422 0c6436b5f1561ecc61f676606fc5b138 https://crt.sh/?id=159974448 47de33e25a3a76e6c86a297994d8fda2 https://crt.sh/?id=161918383 7b7ad30889fcc205af44fe5cb4e3fe1e https://crt.sh/?id=161918400 1f7dd110c03bf97a02356daa3f961108 https://crt.sh/?id=161918411 41027867ef42106d823ad068a04be761 https://crt.sh/?id=161918397 47b08a101c270bda3b7a4a019fb148da https://crt.sh/?id=161918399 669e48d3a81b6d7a06f784facbf1a16b https://crt.sh/?id=161918403 12a4569951d8a375ce346ac731e755d5 https://crt.sh/?id=171387330 25edc88e7a4750d1bc0a8f2294513d19 https://crt.sh/?id=176217303 2f2464e577d755aff46c83386f4e5e76 https://crt.sh/?id=176217371 21bc36f43fe26b7a3949c5904fe207f2 https://crt.sh/?id=197644173 384a843a063189dfe10bd37166851d33 https://crt.sh/?id=197644055 59b0a9fd8031d1c833c8b1f7a4f66594 https://crt.sh/?id=205758807 793804e585e7c5d4cd77327a1c7ca5ec https://crt.sh/?id=205758827 6876d2c366698bcdd4af9f44cdee5349 https://crt.sh/?id=206192605 7e220641d2eaff0f71e435ab96a30e9c https://crt.sh/?id=206192668 27bd147686e1729cd85a79b5a2f8b6e4 https://crt.sh/?id=234189060 2b953afc44dc11d201d45b3b0cbfcf34 https://crt.sh/?id=234189068 4be9d8436a8cb0174f4c84adffd5b449 https://crt.sh/?id=234189072 5c18f40ac307b7a05b56bafa312e6e8d https://crt.sh/?id=239151880 39d52994416898f31bba5d5b4037298c https://crt.sh/?id=239151883 8d3f35dddcc4224aa48214e4cc771a https://crt.sh/?id=239151890 7ffbec9b803c29f6a5e63648ff105c17 https://crt.sh/?id=239151895 532c8f3931780f4c6bb36149b2c1eae1 https://crt.sh/?id=239151912 51cf1d226ac7a1093db173c305d49dc0 https://crt.sh/?id=239151921 4d75bfb99ece05c158796e99082e73a0 https://crt.sh/?id=249659019 34a88f423b8d05ced34e2b1fe5b58d23 https://crt.sh/?id=249659022 09951a2fbb3f3619f6f645dbe5c0085a https://crt.sh/?id=249659026 209bde1eead87762de25afb13d96ec89 https://crt.sh/?id=249659032 3ab7605fa8846406da486654f0139f18 https://crt.sh/?id=254724838 0afdc635cb7c3967755fbb849f3d9126 https://crt.sh/?id=254724914 3d518df386000d52c60079c80018e256 https://crt.sh/?id=254724920 69f97902d9467208817b9a05a88a7802 https://crt.sh/?id=254724861 4c69d21c89e3b2ea38742456ffbd2af4 https://crt.sh/?id=254743440 149748e7dde901eace17be5f54f8d91d https://crt.sh/?id=254743460 23be29eab449b66c5d8aac53c2f4310c https://crt.sh/?id=254743591 69fc165ef71b4cc34a72c6dab0e71f3e https://crt.sh/?id=254744440 529626e82f8cd5ea6fb78c28a4ee085f https://crt.sh/?id=255542880 409820cb1726928651bf21e241a45661 https://crt.sh/?id=255542882 54f2c3ccb2966bf0d0db7afd0d308b77 https://crt.sh/?id=255542885 26afea22f069350640d68db7e9453af7 https://crt.sh/?id=255542884 730969b39689ac5144e1ae647fe9c11b https://crt.sh/?id=255542887 6b838384f940a5fff44eeb11a665e0bc https://crt.sh/?id=255542890 6306b0597ad65adc67209a7a062ceaf6 https://crt.sh/?id=255542894 6ebcec2e85096c7a0d5480d81c085cfb https://crt.sh/?id=255542900 2b55ccd4abd9c41f4094372cbe2ca0a7 https://crt.sh/?id=255542896 7abc80cf9ecfd7646e6305dd1780738e https://crt.sh/?id=255542901 7059e769ee17ea44d47e16d92fb8b5dd https://crt.sh/?id=255542904 6efb8152bf5fb1dab90515308b6593f9 https://crt.sh/?id=255542906 3b2fca615af831161639674f8382ec25 https://crt.sh/?id=255542908 055c96d876e0b63b44c3ca5b9db462f6 https://crt.sh/?id=271004515 04178b78a189712a3009d05d5e6b5aad https://crt.sh/?id=271007351 0f9580aa61a5d3028c9a651361e5011f https://crt.sh/?id=273545463 0b4dd0b423c1db8bb41aaf1282508fa6 https://crt.sh/?id=273557968 073f99a78ac37d4df8714640147c3680 https://crt.sh/?id=273569089 0262767f6a5fbc40be6d99b7f7afeaeb https://crt.sh/?id=311662455 01ec5194252368cf43b56319b15e6ecd https://crt.sh/?id=311662049 05edf706a5ecd2e153f232a646650ac1 https://crt.sh/?id=315857566 07eb34d2edb41ad197356db15e600bf1 https://crt.sh/?id=355192405 020868cc760a04cf68709f785336a56d https://crt.sh/?id=355195641 08bc6e9a3750f57926f4642298faf970 https://crt.sh/?id=367720136 0c6ddae47470cfcb873de522e4b29f48 https://crt.sh/?id=539442119 07c1eca6edf9e9af14511013e3b29788 https://crt.sh/?id=543390138 037a46710cb0a5abcf445a09e54a8391 https://crt.sh/?id=555463042 0d5bd8851788c68069708de46311279d https://crt.sh/?id=584063508 0a561ad65fd0daf2ea24778f0ae8cd81 https://crt.sh/?id=627186062 062a653cb27e9560d73878849da84ca7 https://crt.sh/?id=627187516 0bba6bc08693c75ae661fc735a2fa86f https://crt.sh/?id=628071865 0186aaeffa50106c761aaa235e8f95fb https://crt.sh/?id=628072553 0f68d6bf4cd152c390bfec9fe769c834 https://crt.sh/?id=628086241 03c58b5caade42739f9b87f67506f006 https://crt.sh/?id=628086132 091342de436286e1c65a7824faad0c26 https://crt.sh/?id=628086326 04d4b631028328a9cecf1d2b8230ef2c https://crt.sh/?id=628086509 07800dcc9f245179dacbcfb9bdb00983 https://crt.sh/?id=628086844 0455fbeb4b4364e21a8fcb09c7ed5d42 https://crt.sh/?id=628087032 07bb11db7ec26ba4bc5bf31a605730fd https://crt.sh/?id=628087078 074312ca99ef3faaa061da1ea13132a8 https://crt.sh/?id=628089101 0c4a2ba30f587f96267989d45d1cc3cc https://crt.sh/?id=628091401 0b88ffba6d07971880665180bb2278f5 https://crt.sh/?id=628095660 07385b1cc306aee82d04c90976b453e7 https://crt.sh/?id=628095684 023dc21ba758d2bc8e1d9e5a9e33d531 https://crt.sh/?id=628095714 0e81a9a29d537ef67a6bdcb8362ae277 https://crt.sh/?id=628095754 052afb4ceb71a17f31f03f95fbdb35d8 https://crt.sh/?id=628097298 0414d8632430e5daceecf6cd13a067cd https://crt.sh/?id=628098602 0e7e24c933a4825b70b879cf0a724b86 https://crt.sh/?id=628103967 07a6c1aa421b296d3bdee86987e132fd https://crt.sh/?id=628104117 042f76f5dec774af9e8a04d32d87c0d7 https://crt.sh/?id=628103938 06274053a69fdb145f6b47c02350eba0 https://crt.sh/?id=628103989 0f1115bcddf43e9eb5e9f4e9807ae03b https://crt.sh/?id=628104037 02b98cc2d2a0b81b80885e1ad311a47d https://crt.sh/?id=628104059 05658962ffd26e904a8b5a5cdc7dcbb3 https://crt.sh/?id=628104086 054269143ecbf88d0e09449b8a9e2a32 https://crt.sh/?id=628104136 04a4d134fc10bc9a4495685c3f7f3c12 https://crt.sh/?id=628104186 0f0341483b8748d34cfa95fac4ed10de https://crt.sh/?id=628104205 04d420aad204c248484f89b4573d1006 https://crt.sh/?id=628109469 05a8c697ede7d05db50f7d16b21c0416 https://crt.sh/?id=628109533 0efb50333f937b9224e7b7552bd36463 https://crt.sh/?id=628109552 0f55dd2d9a982fe7051e2ea9cb885d13 https://crt.sh/?id=628109597 0ea8478ba02ead773aaf68f52c904647 https://crt.sh/?id=628109615 03ab6ef2d619af80da784107f5e4a0b4 https://crt.sh/?id=628109630 08c980d703b7df5316aa102b25bcf698 https://crt.sh/?id=628109660 01f832e4d82053c79cbe4590fbf7296a https://crt.sh/?id=628109500 0f90e79864750755fe13c9fab34cb20f https://crt.sh/?id=628109771 0d0e8693db60baec58bede4a59c69f2c https://crt.sh/?id=628109840 0ae13d10861e9d874e06ed6b63b03d0e https://crt.sh/?id=628118332 063b4000cc3d12c2516b5e8170b7d26b https://crt.sh/?id=628118695 033390d7cf24cd0fb34083de601043a9 https://crt.sh/?id=628118288 0ff09413f4d4ec0dde4736006eee5248 https://crt.sh/?id=628118614 09eeb31b911c310f1517befbdf66a075 https://crt.sh/?id=628122694 0aa1998113af273035845e110de26571 https://crt.sh/?id=628126109 09de560f8f93a830b2a37a4475cb76e1 https://crt.sh/?id=628126137 0bcd4dc9ae103ba8049c9b2311eb575b https://crt.sh/?id=628126251 01ea5363afce0292ef8b5d324dd74b16 https://crt.sh/?id=628128395 06f3dc9a78707f1ef624df52085c823d https://crt.sh/?id=628135161 0b5ded168dd58e5e5b45ac1851d0d321 https://crt.sh/?id=628135354 09e188f89a3677a71c6842335d18e042 https://crt.sh/?id=628135522 03eccf9e4f0a1ba212f5893a4b90dcf3 https://crt.sh/?id=628148634 0d089bd5a15e71224b3af53801685388 https://crt.sh/?id=628154756 05f5a3d3b40beac62c7471ba0635a143 https://crt.sh/?id=628154962 09fb0626b789597442763acd2cff332a https://crt.sh/?id=628159694 0e10aa723c15ac54a01aedc84f7455cc https://crt.sh/?id=628160193 09308fa0f31f4e849b4df8dd60582cd5 https://crt.sh/?id=628160424 08c31056a9c75b570bfd74a5c96c25d9 https://crt.sh/?id=628160453 03d7a937080e1f2402f9ab31976eb8b4 https://crt.sh/?id=636167641 0b226d725215de9ef65d12d92d16a7d5 https://crt.sh/?id=664103699 0c6137aab3dc738e8c7cdd2397955596 https://crt.sh/?id=664104899 085869391c7bee642b16adba8a7ededf https://crt.sh/?id=664105913 0e9236ce0bed6bd9dc591a8b41e49101 https://crt.sh/?id=664115209 03195434695c63f728d644f700754219 https://crt.sh/?id=664115140 0d9eef8fd428760b50d51d282fe70fae https://crt.sh/?id=781591132 0f7078145386905a1cd9fc08f1d32f55 https://crt.sh/?id=781591239
Updated•4 years ago
|
Updated•4 years ago
|
Comment 1•4 years ago
|
||
(In reply to Jeremy Rowley from comment #0) > 6. Explanation about how and why the mistakes were made or bugs introduced, > and how they avoided detection until now. > > The freeze period is Dec 16-Jan 13. They are planning the replacement and > will start immediately after the freeze is lifted. Negotiating downtime and > change windows is the delay. > > 7. List of steps CA is taking to resolve the situation and ensure it will > not be repeated. > Still getting this info. They are aware of the need for a better process. > Will post an update when we work something up. I think this remains an important thing to understand, and I'm hoping you have an update here as the deadline approaches. Understanding what steps are or will be taken to ensure that future areas of non-compliance are not revoked in a timely fashion due to "negotiating downtime and change windows" is an important step in ensuring there is meaningful progress. For example, have negotiations begun? How long are they expected to take? Is there a reason the downtime and change window cannot be performed on Jan 14? > 29. April 30, 2018 – Proposal on when all certs will be revoked. There is not sufficient detail, at present, to understand how that conclusion was reached and what information was used to reach it. I'm hoping that for the remaining 74 certificates, DigiCert can share more data about how it (potentially) reached the decision that it was necessary to violate the BRs. Based on this bug alone, it merely appears that "They told us that's when they could do it", which is not particularly inspiring in-and-of-itself.
Updated•4 years ago
|
Reporter | ||
Comment 2•4 years ago
|
||
Based on the conversation on the forum, the post from Wayne, and instruction from Google, our understanding is there is no exception or extension possible and the expectation is that all CAs will revoke the certificates on the date required by the BRs. We hope that the same rules/penalties/expectations will be applied to those CAs who fail to revoke on the required date. Thank you for the discussion. Although we were hoping for more compassionate results, we do appreciate the feedback and clarification on expectations.
Reporter | ||
Comment 3•4 years ago
|
||
Seems there was a mis-communication on the intent of the discussions. We will post an update answering Ryan's questions tomorrow. Please ignore my previous post. Apologies for the confusion.
Comment 4•4 years ago
|
||
Jeremy: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?
Updated•4 years ago
|
Assignee | ||
Comment 5•4 years ago
|
||
Update: All remaining underscore certs for this customer has been revoked as of today (30-April-2019).
Updated•4 years ago
|
Updated•2 months ago
|
Description
•