DigiCert: Underscores - Verizon
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: jeremy.rowley, Assigned: brenda.bernal)
Details
(Whiteboard: [ca-compliance] [ov-misissuance])
User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0 Steps to reproduce: 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. Hard question to answer for a future bug. However, for completeness and the people who don’t follow the CAB Forum here’s the timeline, I figured I'd include all essential notice dates: 1. September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 2. October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance 3. October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal 4. October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 5. October 26, 2018 – Final ballot was proposed. 6. November 2, 2018 – Voting period starts 7. November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs. 8. November 19, 2018 – We first hear of customers not being able to meet the revocation timeline. 9. January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course) 10. April 30, 2018 – Proposal on when all certs will be revoked. If you're talking about prior to this ballot, we were unaware that underscore characters were not allowed. With ballot 202 failing, we weren't sure where that left the industry, especially consider 1034's age, applicability, and what we thought was a goal to secure all websites. Apologies for being incorrect on my reading of the requirements here. I did propose 202 originally to try and clear up the confusion. Customer was given a list of all their impacted certificates on Dec 4, 2018. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. Not sure the best method of presenting this info, but you’ll likely get the most accurate picture with it layered in with the timelines above. I left off the discussion timelines in the CAB forum as that’s all a matter of public record and there is a lot of discussion. Let me know if you want me to add that to this record. 1. September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 2. October 1, 2018 – We cease issuance of underscore characters in case the discussion goes south (obviously it does) 3. October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Bad data leads to only some customers being notified. 4. October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance 5. October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article 6. October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems. 7. October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal 8. October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire. 9. October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 10. October 19, 2018 – Internal discussion to start comms about CAB Forum plan. 11. October 20, 2018 – Second emergency meeting to start comms process. 12. October 24, 2018 – Gather of data on all impacted certs across the different systems 13. October 26, 2018 – Final ballot was proposed. 14. November 1, 2018 – We notice the data is wrong and regather the information. 15. November 2, 2018 – Voting period starts 16. November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs. 20. November 29, 2018 – Posting to Mozilla about concerns with ballot 21. November 29, 2018 – Final comms is dropped about the ballot and its impact. 22. November 30, 2018 – Final internal advisory on issue. 23. December 4, 2018 – Customer given list of certificates and advised to participate in the Mozilla discussion. All exceptions to the revocation date are denied. People to start to escalate to demand that there is an exception process, we just don't know about it yet. 24. December 7, 2018 – Customers engage with Mozilla community 25. December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline 26. December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore. 27. December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs. The goal is to provide better information on the scope of impact. 28. January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course) 29. March 1, 2018 - Proposal on when all certs will be revoked. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements. All certs for this particular entity will be revoked on March 1. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. The problematic certificates are used by third party vendors of the customers to protect systems and device activation. Each update must be coordinated with those vendors. The certificates are used in applications for TLS communication. These applications need to be updated. The difficulty in replacing the impacted certificates is dealing with non-technical vendors. Trust in Mozilla and Chrome is not required for these certs, but they are publicly trusted to support other systems. 5. The complete certificate data for the problematic certificates. Listed below. 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. Replacing the certificates and changing the domain name require identical efforts. The 30 day extension would be great if the other certificates could expire. However, due to the non-technical nature of the audience, replacing a certificate just takes time. The issue with replacement is not technical, it's timing. Replacing the certificates requires interfacing with the third-party vendors and code changes 7. List of steps CA is taking to resolve the situation and ensure it will not be repeated. Working on it. We've got some automated install tools that should be available soon. we're also contemplating how to effectively separate out the MS and Apple ecosystems from Mozilla and Google. Actual results: Cert list: 067AFBE525671158EB03340D6FC132B1 https://crt.sh/?id=1045077524 0C41465D3697BA99DAA2ACA13D92DC91 https://crt.sh/?id=1045077698 08A46FF8555E60DB90B4D107AE9F80BC https://crt.sh/?id=351205723 02E8F33DC6103006F9E4FA4E4EA8F4A8 https://crt.sh/?id=354498065 0B368F7E4D84499078AB4ED859A613F0 https://crt.sh/?id=366737846 061A50A783A59C7D2834BC77B93AA78D https://crt.sh/?id=422238426 0C4DD7389A402D627F8E26A9D3CB46D7 https://crt.sh/?id=498213674 0BA0B8C491E91CA5ECA5699D2D8B4739 https://crt.sh/?id=513721866 04DCB1DE601D537C18E2D8614624C938 https://crt.sh/?id=535649206 0D821AE567D0FB9C3A237A94A9B4CA5A https://crt.sh/?id=537802373 0FA9ED6D96863C78BB65DEC99CA2AE3C https://crt.sh/?id=569935769 04AD6D50086CF298DEFECB012005EC8D https://crt.sh/?id=573713795 09C3E451DAC3FDE83E0ADD802BEB164D https://crt.sh/?id=575705262 0558AFDE53F76875276508085F10115F https://crt.sh/?id=637943959 0C0E94E45BECF4E5187AB2ECBC534CAB https://crt.sh/?id=638262397 065D609A86F8C0ACB6BE33C96FC3FEF2 https://crt.sh/?id=652120712 0D2B1157DE215D3CCFEF0E31015D410B https://crt.sh/?id=691918460 077FAA2B2850A599E4DE87015B364E6B https://crt.sh/?id=719219923 010BB2FC9815E77590530F3B951F70AA https://crt.sh/?id=734089052 0AFDDAD698E5FBDC55B26F7D6AB1504B https://crt.sh/?id=734149351 079F81DC75ED8373E5B512C373244086 https://crt.sh/?id=788281302 0494AF4EA9E9B69829C334E00EF8ED71 https://crt.sh/?id=788281315 0FEC41AC8F2DC5CBD8FC3C7DD00ACD20 https://crt.sh/?id=610912614 0F54D6470D85A3E0E4A0C08C07F9CF5E https://crt.sh/?id=1045077700 0AC0FDCD648D01BA84451DA5F8A13488 https://crt.sh/?id=1045077701 0478DDCB5C5E3FE24E4E40F49A504F7B https://crt.sh/?id=1045077699 083E078B38FD0661FD35104F6A606D41 https://crt.sh/?id=1045077702 01CA43C625454D55D561AE9E2C67D81C https://crt.sh/?id=1045077715 0F3A532D7086E77032AC4E3B012FDC9F https://crt.sh/?id=1045077715 08220DE6D21F721B93F0FC67250C1D87 https://crt.sh/?id=351349397 08AE5D6CB10820E9B91206DCBB821F0A https://crt.sh/?id=362641010 0E40585F8F1862A93637E95F38E62211 https://crt.sh/?id=375101861 0E33BE9ED73661B0A6300EA0A728CAB7 https://crt.sh/?id=376863079 01B818CB31C9756C2F69D3C9A36191D1 https://crt.sh/?id=399472473 0772E175E61BE71ACB381AC03C38CE34 https://crt.sh/?id=399536609 0760FFDAB9EC7AB36B9819CFB41CAD25 https://crt.sh/?id=420041491 0883FE207E4650BAAA6F32FDDB5425EF https://crt.sh/?id=420531356 0B0BC734834BCF3113B901316C8B548D https://crt.sh/?id=420530758 050FE4A5BE645EE2AF3F6A7FFE9ABC75 https://crt.sh/?id=420529825 03007FB55296B17C35ED9DF75B5A94CB https://crt.sh/?id=449515359 0DCB58BDDE9A83D79A44BD52D42FC98F https://crt.sh/?id=451795063 0B3B6114B8B734B2855490537BEE14B4 https://crt.sh/?id=451786962 07495DB7238C3F5AA42B5F80E08B20CD https://crt.sh/?id=478578166 015BC8968615B047CD987745DF60FA3E https://crt.sh/?id=478581209 0F902C070A820025DBE5A93C1BC78D47 https://crt.sh/?id=483729843 0408DED6AC460C428E480AEC252D7E9F https://crt.sh/?id=493665442 07757D0D169613898594E6B949FA8282 https://crt.sh/?id=493663590 09DC78D330CA91C20DA8BE46142EF7AD https://crt.sh/?id=511647166 0BC1FA57FFD3634264170D4404C13A36 https://crt.sh/?id=511627129 0D6121F044475774660F907B617C4876 https://crt.sh/?id=511636098 0C54ABF3A6B2E6655C2BBDD2A28D1A3F https://crt.sh/?id=511637988 0DD59B29B979BDF5E76CD8DBBFB23491 https://crt.sh/?id=511683455 08A972B97D34E207AD388A7CA6C45483 https://crt.sh/?id=505663120 0145BB30C9ACFFEAF57C0623B4BD22DB https://crt.sh/?id=505661860 03574CA95D1BEDE56FD83B199C1F5AA9 https://crt.sh/?id=505679795 0670675F5DE395129FF46A84FEEE48FC https://crt.sh/?id=505661329 0D40EE2BF6C1B5A7DF23AA1429EBE75D https://crt.sh/?id=512127075 028EA33B40198C7570A74E090618B5FF https://crt.sh/?id=512023763 086CE24F6A563D9E6076A2A14B17F628 https://crt.sh/?id=512126430 06B645C5B9C7E8D4034F594D9B186FF1 https://crt.sh/?id=512126291 010DD25E03A28A0D0B58CDB154CDCA05 https://crt.sh/?id=542280782 065702DC2F497650BBB8B6F19A98E122 https://crt.sh/?id=542280937 03FCA3BAF9C7A08D7D035525E62A8C31 https://crt.sh/?id=597040329 0BC9D97B4B65C4DD561E899F06BFF116 https://crt.sh/?id=597040465 0264156635FF0E43E24FD7BC2618E389 https://crt.sh/?id=597040735 0F7D602239CE4EBCA0F25F31EDE768A4 https://crt.sh/?id=597087403 080E38EF925792AA66CECA8793BB68C5 https://crt.sh/?id=597087412 07B16491DEA7171678F4E7FB29F2ADF8 https://crt.sh/?id=597040086 02967003DCD94F0138417501A9C57A67 https://crt.sh/?id=597039993 04ABCC62A66894A65A8CCAFC8A572D46 https://crt.sh/?id=597052850 05B595304B36A304EC6F6C7DCB1E96B8 https://crt.sh/?id=605210070 0EFFFF3F2E12F4AB1DA0046B48CCE93C https://crt.sh/?id=605317638 0D406D716D2BC85DA23EEB6109D19CD9 https://crt.sh/?id=607259740 085C409DB4A14FE3D73824E7BAD95CFF https://crt.sh/?id=607273041 02AD6E2A3EBADF036AD96D3A194769F9 https://crt.sh/?id=607425561 0D4E0C7DE642D914E288A04F42BDF2EA https://crt.sh/?id=610065291 0B7936D3FCC231F6E1BC94D0FCFB033C https://crt.sh/?id=610075800 0C1A2DB43860594E141945BF2598302A https://crt.sh/?id=610191557 02FE113924803097BBB24ED1B64F3784 https://crt.sh/?id=610190877 0D40520E24D04E32248BD48CCF1C0006 https://crt.sh/?id=610190902 0FEA5D2AD6EE65F98C3B8E7C7446953C https://crt.sh/?id=610189945 07364DA01FBB74519A380B03C4863919 https://crt.sh/?id=610181126 07A84D279264DB40AAB7A4DDE25E2892 https://crt.sh/?id=610181141 09D087EEEC017B03DC81E8D23FB02C12 https://crt.sh/?id=610169803 06C92B771658BFE4916ED9C878F8D9A5 https://crt.sh/?id=610170471 03B7D2804D0607C26964C5F550D6D785 https://crt.sh/?id=610173345 015C3B601EEAA80D5B99F2480D48BD62 https://crt.sh/?id=610175122 04554969481876EE2DFE92FDBD807EEF https://crt.sh/?id=610700027 0642EE88E2C31A4F9E6D4934B8840FCE https://crt.sh/?id=610699976 0E88B0FF1269FB2813AFE34E163E0F3D https://crt.sh/?id=610699897 0E2F87E964E20ABC037D2AF967641BFB https://crt.sh/?id=610699677 0B94DDF23AA64A0DA2F3D1A0F02E8D0C https://crt.sh/?id=610699571 02BE307EB550BF5F3ABE654EA0BD480D https://crt.sh/?id=621506079 0CA85E42A23415D0A0638AA9B9DCA32E https://crt.sh/?id=627771950 060BFA3CC44DA0FACB90576E133393F6 https://crt.sh/?id=635874783 0585A4F834E91C434DABDAC06C512277 https://crt.sh/?id=637945217 0301BB793BD668997D240FF7BFD99B1C https://crt.sh/?id=648725572 01FE81CC6C7294781FBB1DD84C60035B https://crt.sh/?id=649598159 08390CD8B7D82A329F2D4ACE2FA72AC8 https://crt.sh/?id=649598675 05036BB34208AABE081336D84BD8B83E https://crt.sh/?id=649598631 0640C1E86A429695A8B5C8F886161C00 https://crt.sh/?id=649611893 0E9C8B07A713F147003FE4C3797D37F3 https://crt.sh/?id=649611859 055481E946DEFC66FDCD9A4C456EF688 https://crt.sh/?id=649611710 0E8166A39A8CC12D30C53C007D58B0F6 https://crt.sh/?id=649670650 084FB73D91AFF21376030FD43C202802 https://crt.sh/?id=656543966 082CF1BAD1EB25DD62BE997617EF4468 https://crt.sh/?id=672162628 063BF57AE4EB1FFEF0FAB3056434C08C https://crt.sh/?id=672918782 0DE6CA73C25836617F49FBE3B0611B33 https://crt.sh/?id=672918543 04E9C8B38AD23D4FEECD877486AAD3AF https://crt.sh/?id=672932669 0BE958603858E1597BB74AE8C57A8EF3 https://crt.sh/?id=672932702 0D31D37EEC65266E0B8D1297208C1E4F https://crt.sh/?id=673155313 0826E1FE89E1DE89679F7E3F5570959B https://crt.sh/?id=673155328 0A86DBC1970D2975BF3F8C1A1FD8962E https://crt.sh/?id=673155372 01B5B44BFC2A0730956577904F07BA4E https://crt.sh/?id=673155916 072474024F3C3CE5025506B4D794E32F https://crt.sh/?id=684120952 0970E19F0CF5D524789EE0259EE72A9A https://crt.sh/?id=715447405 0590F842D8B5F2DE2D8AA0517C86EEBF https://crt.sh/?id=720784445 01DED16CA8321300BE1DADEF7D47B5DA https://crt.sh/?id=740171045 0B73F93F19ED45F4C0DD0890289CD3BF https://crt.sh/?id=742516342 09DAA5BA58C72FDFD7B7D92072574FF6 https://crt.sh/?id=775374697 0495169A0E9BDD30F45AE6E50F90457C https://crt.sh/?id=785696736
Updated•5 years ago
|
Comment 1•5 years ago
|
||
Just as an idea: Maybe you could perform together with your customer a triage and start with the revokation of those certificates which are not production relevant? E.g. https://crt.sh/?id=512126430 seems to be (at least acc. to the CN) non-productive.
Comment 2•5 years ago
|
||
In addition to Rufus' remark, which seems useful to understanding what triage steps DigiCert has performed with Verizon Data Services, I think it's important to further unpack this statement: > The certificates are used in applications for TLS communication. These applications need to be updated. Wayne mentioned this in the context of https://groups.google.com/d/msg/mozilla.dev.security.policy/zB5N0PT6-b4/ObbLh1BGAwAJ While I understand that "trust in Mozilla and Chrome is not required for these certs", they do come from the "DigiCert SHA2 Secure Server CA". In seeking to understand how DigiCert is weighing the necessity of this, it'd be useful to understand to what extent DigiCert supports this use case for publicly trusted certificates, and what documentation or advice is provided for Subscribers if this is seen as valid, to support DigiCert and the Subscriber meeting their obligations under the Subscriber Agreement and CPS. Put differently: If this use case is not supported, and DigiCert decides not to revoke, what steps is it taking to both clarify this position for all of its Subscribers and to ensure that, in the future, it does consistently revoke for this use case? If the use case is supported, what rubric does DigiCert use when determining whether or not to violate the BRs in support of this use case, and how were the facts of this situation evaluated against that rubric? Basically, more information is needed here to understand the decision making process used to determine whether or not this is a reasonable request and/or how impact is assessed and mitigated.
Reporter | ||
Comment 3•5 years ago
|
||
Production relevant doesn't necessarily mean easier to revoke. We are working with the customers to identify a revocation timeline for each certificate, not just the global date.
Most certificates come from publicly trusted roots. We offer the private option in the same platform as our public cert offering, which means people can order either.
THere's lots of documentation on private vs. public. eg. https://www.digicert.com/pki/?gclid=Cj0KCQiAjszhBRDgARIsAH8KgveU6ra7tvEzuMVAYyB6qf6VM6Ietz26q4_pC7veEoALZq9Sjd9zcFYaAlgIEALw_wcB&&ef_id=Cj0KCQiAjszhBRDgARIsAH8KgveU6ra7tvEzuMVAYyB6qf6VM6Ietz26q4_pC7veEoALZq9Sjd9zcFYaAlgIEALw_wcB:G:s&s_kwcid=AL!6100!3!296995531423!b!!g!!%2Bdigicert%20%2Bpki
and
https://www.digicert.com/private-pki/
We leave it up to the customer to decide which certificate is appropriate. For publicly-trusted certificates, the agreement calls out the requirement to comply with the BRs and the revocation timelines.
I'm not sure I understand the question about the rubric.
We've never intentionally violate the BRs. This would be a first. There is no rubric as we've never needed one. The facts in this situation were unique in the size of the customers, the systems supported by underscores, and the timing of the change. Because it was unique, we wanted to raise it on the CAB forum. The deprecation of underscores is different than something like SHA1 simply because it occurred over a holiday without a clear reason behind the urgency.
Updated•5 years ago
|
Reporter | ||
Comment 4•5 years ago
|
||
Based on the conversation on the forum, the post from Wayne, and instruction from Google, our understanding is there is no exception or extension possible and the expectation is that all CAs will revoke the certificates on the date required by the BRs. We hope that the same rules/penalties/expectations will be applied to those CAs who fail to revoke on the required date. Thank you for the discussion. Although we were hoping for more compassionate results, we do appreciate the feedback and clarification on expectations.
Reporter | ||
Comment 5•5 years ago
|
||
Seems there was a mis-communication on the intent of the discussions. We will post an update answering Ryan's questions tomorrow. Please ignore my previous post.
Assignee | ||
Comment 6•5 years ago
|
||
To complete the list of certs in scope, below are the ones for the wireless portion of the business:
https://crt.sh/?id=72874948
https://crt.sh/?id=78168649
https://crt.sh/?id=80809816
https://crt.sh/?id=82848289
https://crt.sh/?id=83935732
https://crt.sh/?id=83935729
https://crt.sh/?id=85986717
https://crt.sh/?id=89221762
https://crt.sh/?id=89288372
https://crt.sh/?id=91635372
https://crt.sh/?id=92938121
https://crt.sh/?id=93441845
https://crt.sh/?id=96401782
https://crt.sh/?id=97292524
https://crt.sh/?id=97347120
https://crt.sh/?id=99730317
https://crt.sh/?id=103207949
https://crt.sh/?id=103563466
https://crt.sh/?id=104142725
https://crt.sh/?id=110052332
https://crt.sh/?id=110052331
https://crt.sh/?id=110052330
Comment 7•5 years ago
|
||
Brenda: I'm just trying to make sure I'm not missing something. It seems the certs listed in Comment #6 are not part of the original discussion from comment #0 - is that correct? Could you help me understand both the addition and the impact to the timelines this represents?
Assignee | ||
Comment 8•5 years ago
|
||
Hi Ryan, The first list of certs at the top were from the company's data services line of business, whereas the second list is in addition to that list, for their wireless business. In both cases, they are requesting for an extension of time to ensure all certs affected can be properly replaced and tested. The request is for time until March 1, 2019 (correction on the 2018 date). Their internal team has been diligently working through the replacements and change control process after their year-end freeze period, which just concluded.
Comment 9•5 years ago
|
||
Jeremy: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?
Assignee | ||
Comment 10•5 years ago
|
||
Hi Ryan, I will be responding to provide updates on the underscore incidents. I can confirm that an incident has occurred and the details provided are accurate to the best of our knowledge. Our planned extension to revoke the remaining certificates (listed above) is 1-March-2019 (corrected the 2018 date in comment 8). We will provide periodic updates as progress is made.
Updated•5 years ago
|
Assignee | ||
Comment 11•5 years ago
|
||
Update: the remaining underscore certs were revoked as of 01-March-2019.
Comment 12•5 years ago
|
||
It appears that all questions have been answered and remediation has been completed.
Updated•1 year ago
|
Updated•1 year ago
|
Description
•