Closed Bug 1516545 Opened 5 years ago Closed 5 years ago

DigiCert: Underscores - Verizon

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: brenda.bernal)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Hard question to answer for a future bug. However, for completeness and the people who don’t follow the CAB Forum here’s the timeline, I figured I'd include all essential notice dates:

1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
3.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
4.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
5.	October 26, 2018 – Final ballot was proposed. 
6.	November 2, 2018 – Voting period starts
7.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
8.	November 19, 2018 – We first hear of customers not being able to meet the revocation timeline.
9.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
10.	April 30, 2018 – Proposal on when all certs will be revoked.

If you're talking about prior to this ballot, we were unaware that underscore characters were not allowed. With ballot 202 failing, we weren't sure where that left the industry, especially consider 1034's age, applicability, and what we thought was a goal to secure all websites. Apologies for being incorrect on my reading of the requirements here. I did propose 202 originally to try and clear up the confusion. 

Customer was given a list of all their impacted certificates on Dec 4, 2018.

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 
Not sure the best method of presenting this info, but you’ll likely get the most accurate picture with it layered in with the timelines above. I left off the discussion timelines in the CAB forum as that’s all a matter of public record and there is a lot of discussion. Let me know if you want me to add that to this record.
1.	September 5, 2018 - the issue is raised by the browsers on the CA/Browser Forum. 
2.	October 1, 2018 – We cease issuance of underscore characters in case the discussion goes south (obviously it does) 
3.	October 2, 2018 – We notify customers that the browsers are raising an issue with underscores. Bad data leads to only some customers being notified. 
4.	October 10, 2018 – The CAB Forum discussions on the validation working group indicated that the browsers believed this was mis-issuance
5.	October 10, 2018 – Internal advisory sent that this is picking up speed and external comms provided in KB article
6.	October 11, 2018 – Discussion with customers about potential impact. Turns out they are required for certain IBM systems.
7.	October 16, 201 – Tim reports back on the status of the Shanghai meeting. This is when we first know of the proposal
8.	October 17, 2018 – Internal discussion about whether we allow underscore character renewals and whether the ballot is likely to pass. We decide it is but are hoping existing certs will be allowed to expire.
9.	October 19, 2018 – Ballot was proposed by Wayne to the validation working group. This is the first we aware that the certs may require revocation. Note the revocation date was still being debated. 
10.	October 19, 2018 – Internal discussion to start comms about CAB Forum plan.
11.	October 20, 2018 – Second emergency meeting to start comms process.
12.	October 24, 2018 – Gather of data on all impacted certs across the different systems
13.	October 26, 2018 – Final ballot was proposed. 
14.	November 1, 2018 – We notice the data is wrong and regather the information.  
15.	November 2, 2018 – Voting period starts
16.	November 9, 2018 – Voting period ends. This is when we first know there is a requirement in the CAB Forum to revoke the certs.
20.	November 29, 2018 – Posting to Mozilla about concerns with ballot
21.	November 29, 2018 – Final comms is dropped about the ballot and its impact. 
22.	November 30, 2018 – Final internal advisory on issue.
23.	December 4, 2018 – Customer given list of certificates and advised to participate in the Mozilla discussion. All exceptions to the revocation date are denied. People to start to escalate to demand that there is an exception process, we just don't know about it yet.  
24.	December 7, 2018 – Customers engage with Mozilla community
25.	December 5, 2018 – Daily calls start to try and identify why people can’t migrate by the required timeline
26.	December 12, 2018 – Question about scope asked of Mozilla. Does legacy Symantec really need to be replaced? They aren’t trusted by Mozilla anymore.
27.	December 19, 2018 – Post of future incident report to start discussion on what will happen if we don’t revoke the certs.  The goal is to provide better information on the scope of impact.
28.	January 15, 2018 – First time we will be in non-compliance (assuming we don’t revoke all the certs of course)
29.	March 1, 2018 - Proposal on when all certs will be revoked.
 
3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certs with underscore characters on Oct 1. We re-enabled 30 day certificates per the ballot for any customers that can use that option. We found that exactly no customers can use that option. We will shut down the 30 day certs per the ballot requirements.  All certs for this particular entity will be revoked on March 1.
 
4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 

The problematic certificates are used by third party vendors of the customers to protect systems and device activation. Each update must be coordinated with those vendors. The certificates are used in applications for TLS communication. These applications need to be updated. The difficulty in replacing the impacted certificates is dealing with non-technical vendors. Trust in Mozilla and Chrome is not required for these certs, but they are publicly trusted to support other systems. 

5.	The complete certificate data for the problematic certificates. 
Listed below. 

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Replacing the certificates and changing the domain name require identical efforts. The 30 day extension would be great if the other certificates could expire. However, due to the non-technical nature of the audience, replacing a certificate just takes time. The issue with replacement is not technical, it's timing. Replacing the certificates requires interfacing with the third-party vendors and code changes

7.	 List of steps CA is taking to resolve the situation and ensure it will not be repeated. 
Working on it. We've got some automated install tools that should be available soon. we're also contemplating how to effectively separate out the MS and Apple ecosystems from Mozilla and Google. 


Actual results:

Cert list:
067AFBE525671158EB03340D6FC132B1	https://crt.sh/?id=1045077524
0C41465D3697BA99DAA2ACA13D92DC91	https://crt.sh/?id=1045077698
08A46FF8555E60DB90B4D107AE9F80BC	https://crt.sh/?id=351205723
02E8F33DC6103006F9E4FA4E4EA8F4A8	https://crt.sh/?id=354498065
0B368F7E4D84499078AB4ED859A613F0	https://crt.sh/?id=366737846
061A50A783A59C7D2834BC77B93AA78D	https://crt.sh/?id=422238426
0C4DD7389A402D627F8E26A9D3CB46D7	https://crt.sh/?id=498213674
0BA0B8C491E91CA5ECA5699D2D8B4739	https://crt.sh/?id=513721866
04DCB1DE601D537C18E2D8614624C938	https://crt.sh/?id=535649206
0D821AE567D0FB9C3A237A94A9B4CA5A	https://crt.sh/?id=537802373
0FA9ED6D96863C78BB65DEC99CA2AE3C	https://crt.sh/?id=569935769
04AD6D50086CF298DEFECB012005EC8D	https://crt.sh/?id=573713795
09C3E451DAC3FDE83E0ADD802BEB164D	https://crt.sh/?id=575705262
0558AFDE53F76875276508085F10115F	https://crt.sh/?id=637943959
0C0E94E45BECF4E5187AB2ECBC534CAB	https://crt.sh/?id=638262397
065D609A86F8C0ACB6BE33C96FC3FEF2	https://crt.sh/?id=652120712
0D2B1157DE215D3CCFEF0E31015D410B	https://crt.sh/?id=691918460
077FAA2B2850A599E4DE87015B364E6B	https://crt.sh/?id=719219923
010BB2FC9815E77590530F3B951F70AA	https://crt.sh/?id=734089052
0AFDDAD698E5FBDC55B26F7D6AB1504B	https://crt.sh/?id=734149351
079F81DC75ED8373E5B512C373244086	https://crt.sh/?id=788281302
0494AF4EA9E9B69829C334E00EF8ED71	https://crt.sh/?id=788281315
0FEC41AC8F2DC5CBD8FC3C7DD00ACD20	https://crt.sh/?id=610912614
0F54D6470D85A3E0E4A0C08C07F9CF5E	https://crt.sh/?id=1045077700
0AC0FDCD648D01BA84451DA5F8A13488	https://crt.sh/?id=1045077701
0478DDCB5C5E3FE24E4E40F49A504F7B	https://crt.sh/?id=1045077699
083E078B38FD0661FD35104F6A606D41	https://crt.sh/?id=1045077702
01CA43C625454D55D561AE9E2C67D81C	https://crt.sh/?id=1045077715
0F3A532D7086E77032AC4E3B012FDC9F	https://crt.sh/?id=1045077715
08220DE6D21F721B93F0FC67250C1D87	https://crt.sh/?id=351349397
08AE5D6CB10820E9B91206DCBB821F0A	https://crt.sh/?id=362641010
0E40585F8F1862A93637E95F38E62211	https://crt.sh/?id=375101861
0E33BE9ED73661B0A6300EA0A728CAB7	https://crt.sh/?id=376863079
01B818CB31C9756C2F69D3C9A36191D1	https://crt.sh/?id=399472473
0772E175E61BE71ACB381AC03C38CE34	https://crt.sh/?id=399536609
0760FFDAB9EC7AB36B9819CFB41CAD25	https://crt.sh/?id=420041491
0883FE207E4650BAAA6F32FDDB5425EF	https://crt.sh/?id=420531356
0B0BC734834BCF3113B901316C8B548D	https://crt.sh/?id=420530758
050FE4A5BE645EE2AF3F6A7FFE9ABC75	https://crt.sh/?id=420529825
03007FB55296B17C35ED9DF75B5A94CB	https://crt.sh/?id=449515359
0DCB58BDDE9A83D79A44BD52D42FC98F	https://crt.sh/?id=451795063
0B3B6114B8B734B2855490537BEE14B4	https://crt.sh/?id=451786962
07495DB7238C3F5AA42B5F80E08B20CD	https://crt.sh/?id=478578166
015BC8968615B047CD987745DF60FA3E	https://crt.sh/?id=478581209
0F902C070A820025DBE5A93C1BC78D47	https://crt.sh/?id=483729843
0408DED6AC460C428E480AEC252D7E9F	https://crt.sh/?id=493665442
07757D0D169613898594E6B949FA8282	https://crt.sh/?id=493663590
09DC78D330CA91C20DA8BE46142EF7AD	https://crt.sh/?id=511647166
0BC1FA57FFD3634264170D4404C13A36	https://crt.sh/?id=511627129
0D6121F044475774660F907B617C4876	https://crt.sh/?id=511636098
0C54ABF3A6B2E6655C2BBDD2A28D1A3F	https://crt.sh/?id=511637988
0DD59B29B979BDF5E76CD8DBBFB23491	https://crt.sh/?id=511683455
08A972B97D34E207AD388A7CA6C45483	https://crt.sh/?id=505663120
0145BB30C9ACFFEAF57C0623B4BD22DB	https://crt.sh/?id=505661860
03574CA95D1BEDE56FD83B199C1F5AA9	https://crt.sh/?id=505679795
0670675F5DE395129FF46A84FEEE48FC	https://crt.sh/?id=505661329
0D40EE2BF6C1B5A7DF23AA1429EBE75D	https://crt.sh/?id=512127075
028EA33B40198C7570A74E090618B5FF	https://crt.sh/?id=512023763
086CE24F6A563D9E6076A2A14B17F628	https://crt.sh/?id=512126430
06B645C5B9C7E8D4034F594D9B186FF1	https://crt.sh/?id=512126291
010DD25E03A28A0D0B58CDB154CDCA05	https://crt.sh/?id=542280782
065702DC2F497650BBB8B6F19A98E122	https://crt.sh/?id=542280937
03FCA3BAF9C7A08D7D035525E62A8C31	https://crt.sh/?id=597040329
0BC9D97B4B65C4DD561E899F06BFF116	https://crt.sh/?id=597040465
0264156635FF0E43E24FD7BC2618E389	https://crt.sh/?id=597040735
0F7D602239CE4EBCA0F25F31EDE768A4	https://crt.sh/?id=597087403
080E38EF925792AA66CECA8793BB68C5	https://crt.sh/?id=597087412
07B16491DEA7171678F4E7FB29F2ADF8	https://crt.sh/?id=597040086
02967003DCD94F0138417501A9C57A67	https://crt.sh/?id=597039993
04ABCC62A66894A65A8CCAFC8A572D46	https://crt.sh/?id=597052850
05B595304B36A304EC6F6C7DCB1E96B8	https://crt.sh/?id=605210070
0EFFFF3F2E12F4AB1DA0046B48CCE93C	https://crt.sh/?id=605317638
0D406D716D2BC85DA23EEB6109D19CD9	https://crt.sh/?id=607259740
085C409DB4A14FE3D73824E7BAD95CFF	https://crt.sh/?id=607273041
02AD6E2A3EBADF036AD96D3A194769F9	https://crt.sh/?id=607425561
0D4E0C7DE642D914E288A04F42BDF2EA	https://crt.sh/?id=610065291
0B7936D3FCC231F6E1BC94D0FCFB033C	https://crt.sh/?id=610075800
0C1A2DB43860594E141945BF2598302A	https://crt.sh/?id=610191557
02FE113924803097BBB24ED1B64F3784	https://crt.sh/?id=610190877
0D40520E24D04E32248BD48CCF1C0006	https://crt.sh/?id=610190902
0FEA5D2AD6EE65F98C3B8E7C7446953C	https://crt.sh/?id=610189945
07364DA01FBB74519A380B03C4863919	https://crt.sh/?id=610181126
07A84D279264DB40AAB7A4DDE25E2892	https://crt.sh/?id=610181141
09D087EEEC017B03DC81E8D23FB02C12	https://crt.sh/?id=610169803
06C92B771658BFE4916ED9C878F8D9A5	https://crt.sh/?id=610170471
03B7D2804D0607C26964C5F550D6D785	https://crt.sh/?id=610173345
015C3B601EEAA80D5B99F2480D48BD62	https://crt.sh/?id=610175122
04554969481876EE2DFE92FDBD807EEF	https://crt.sh/?id=610700027
0642EE88E2C31A4F9E6D4934B8840FCE	https://crt.sh/?id=610699976
0E88B0FF1269FB2813AFE34E163E0F3D	https://crt.sh/?id=610699897
0E2F87E964E20ABC037D2AF967641BFB	https://crt.sh/?id=610699677
0B94DDF23AA64A0DA2F3D1A0F02E8D0C	https://crt.sh/?id=610699571
02BE307EB550BF5F3ABE654EA0BD480D	https://crt.sh/?id=621506079
0CA85E42A23415D0A0638AA9B9DCA32E	https://crt.sh/?id=627771950
060BFA3CC44DA0FACB90576E133393F6	https://crt.sh/?id=635874783
0585A4F834E91C434DABDAC06C512277	https://crt.sh/?id=637945217
0301BB793BD668997D240FF7BFD99B1C	https://crt.sh/?id=648725572
01FE81CC6C7294781FBB1DD84C60035B	https://crt.sh/?id=649598159
08390CD8B7D82A329F2D4ACE2FA72AC8	https://crt.sh/?id=649598675
05036BB34208AABE081336D84BD8B83E	https://crt.sh/?id=649598631
0640C1E86A429695A8B5C8F886161C00	https://crt.sh/?id=649611893
0E9C8B07A713F147003FE4C3797D37F3	https://crt.sh/?id=649611859
055481E946DEFC66FDCD9A4C456EF688	https://crt.sh/?id=649611710
0E8166A39A8CC12D30C53C007D58B0F6	https://crt.sh/?id=649670650
084FB73D91AFF21376030FD43C202802	https://crt.sh/?id=656543966
082CF1BAD1EB25DD62BE997617EF4468	https://crt.sh/?id=672162628
063BF57AE4EB1FFEF0FAB3056434C08C	https://crt.sh/?id=672918782
0DE6CA73C25836617F49FBE3B0611B33	https://crt.sh/?id=672918543
04E9C8B38AD23D4FEECD877486AAD3AF	https://crt.sh/?id=672932669
0BE958603858E1597BB74AE8C57A8EF3	https://crt.sh/?id=672932702
0D31D37EEC65266E0B8D1297208C1E4F	https://crt.sh/?id=673155313
0826E1FE89E1DE89679F7E3F5570959B	https://crt.sh/?id=673155328
0A86DBC1970D2975BF3F8C1A1FD8962E	https://crt.sh/?id=673155372
01B5B44BFC2A0730956577904F07BA4E	https://crt.sh/?id=673155916
072474024F3C3CE5025506B4D794E32F	https://crt.sh/?id=684120952
0970E19F0CF5D524789EE0259EE72A9A	https://crt.sh/?id=715447405
0590F842D8B5F2DE2D8AA0517C86EEBF	https://crt.sh/?id=720784445
01DED16CA8321300BE1DADEF7D47B5DA	https://crt.sh/?id=740171045
0B73F93F19ED45F4C0DD0890289CD3BF	https://crt.sh/?id=742516342
09DAA5BA58C72FDFD7B7D92072574FF6	https://crt.sh/?id=775374697
0495169A0E9BDD30F45AE6E50F90457C	https://crt.sh/?id=785696736
Assignee: wthayer → jeremy.rowley
Summary: DigiCert Underscores - A large data company that you definitely can't tell the name of by looking at the certs → DigiCert: Underscores - A large data company that you definitely can't tell the name of by looking at the certs
Whiteboard: [ca-compliance]
Just as an idea: Maybe you could perform together with your customer a triage and start with the revokation of those certificates which are not production relevant? E.g. https://crt.sh/?id=512126430 seems to be (at least acc. to the CN) non-productive.
In addition to Rufus' remark, which seems useful to understanding what triage steps DigiCert has performed with Verizon Data Services, I think it's important to further unpack this statement:

> The certificates are used in applications for TLS communication. These applications need to be updated.

Wayne mentioned this in the context of https://groups.google.com/d/msg/mozilla.dev.security.policy/zB5N0PT6-b4/ObbLh1BGAwAJ

While I understand that "trust in Mozilla and Chrome is not required for these certs", they do come from the "DigiCert SHA2 Secure Server CA".

In seeking to understand how DigiCert is weighing the necessity of this, it'd be useful to understand to what extent DigiCert supports this use case for publicly trusted certificates, and what documentation or advice is provided for Subscribers if this is seen as valid, to support DigiCert and the Subscriber meeting their obligations under the Subscriber Agreement and CPS.

Put differently: If this use case is not supported, and DigiCert decides not to revoke, what steps is it taking to both clarify this position for all of its Subscribers and to ensure that, in the future, it does consistently revoke for this use case? If the use case is supported, what rubric does DigiCert use when determining whether or not to violate the BRs in support of this use case, and how were the facts of this situation evaluated against that rubric?

Basically, more information is needed here to understand the decision making process used to determine whether or not this is a reasonable request and/or how impact is assessed and mitigated.
Flags: needinfo?(jeremy.rowley)

Production relevant doesn't necessarily mean easier to revoke. We are working with the customers to identify a revocation timeline for each certificate, not just the global date.

Most certificates come from publicly trusted roots. We offer the private option in the same platform as our public cert offering, which means people can order either.

THere's lots of documentation on private vs. public. eg. https://www.digicert.com/pki/?gclid=Cj0KCQiAjszhBRDgARIsAH8KgveU6ra7tvEzuMVAYyB6qf6VM6Ietz26q4_pC7veEoALZq9Sjd9zcFYaAlgIEALw_wcB&&ef_id=Cj0KCQiAjszhBRDgARIsAH8KgveU6ra7tvEzuMVAYyB6qf6VM6Ietz26q4_pC7veEoALZq9Sjd9zcFYaAlgIEALw_wcB:G:s&s_kwcid=AL!6100!3!296995531423!b!!g!!%2Bdigicert%20%2Bpki

and

https://www.digicert.com/private-pki/

We leave it up to the customer to decide which certificate is appropriate. For publicly-trusted certificates, the agreement calls out the requirement to comply with the BRs and the revocation timelines.

I'm not sure I understand the question about the rubric.

We've never intentionally violate the BRs. This would be a first. There is no rubric as we've never needed one. The facts in this situation were unique in the size of the customers, the systems supported by underscores, and the timing of the change. Because it was unique, we wanted to raise it on the CAB forum. The deprecation of underscores is different than something like SHA1 simply because it occurred over a holiday without a clear reason behind the urgency.

Flags: needinfo?(jeremy.rowley)
Summary: DigiCert: Underscores - A large data company that you definitely can't tell the name of by looking at the certs → DigiCert: Underscores - Verizon

Based on the conversation on the forum, the post from Wayne, and instruction from Google, our understanding is there is no exception or extension possible and the expectation is that all CAs will revoke the certificates on the date required by the BRs. We hope that the same rules/penalties/expectations will be applied to those CAs who fail to revoke on the required date. Thank you for the discussion. Although we were hoping for more compassionate results, we do appreciate the feedback and clarification on expectations.

Seems there was a mis-communication on the intent of the discussions. We will post an update answering Ryan's questions tomorrow. Please ignore my previous post.

Brenda: I'm just trying to make sure I'm not missing something. It seems the certs listed in Comment #6 are not part of the original discussion from comment #0 - is that correct? Could you help me understand both the addition and the impact to the timelines this represents?

Flags: needinfo?(brenda.bernal)

Hi Ryan, The first list of certs at the top were from the company's data services line of business, whereas the second list is in addition to that list, for their wireless business. In both cases, they are requesting for an extension of time to ensure all certs affected can be properly replaced and tested. The request is for time until March 1, 2019 (correction on the 2018 date). Their internal team has been diligently working through the replacements and change control process after their year-end freeze period, which just concluded.

Flags: needinfo?(brenda.bernal)

Jeremy: Can you confirm whether an incident occurred and ensure that all of the details of affected certs are accurate?

Flags: needinfo?(jeremy.rowley)

Hi Ryan, I will be responding to provide updates on the underscore incidents. I can confirm that an incident has occurred and the details provided are accurate to the best of our knowledge. Our planned extension to revoke the remaining certificates (listed above) is 1-March-2019 (corrected the 2018 date in comment 8). We will provide periodic updates as progress is made.

Assignee: jeremy.rowley → brenda.bernal
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Update: the remaining underscore certs were revoked as of 01-March-2019.

It appears that all questions have been answered and remediation has been completed.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(jeremy.rowley)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.