1651447 - GlobalSign: Failure to revoke noncompliant ICA within 7 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Arvid Vermote]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0

Comment 1

[Arvid Vermote]

Creating this bug to capture GlobalSign not able to revoke all CA affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 within the 7 days time frame as set forth by section #4.9.1.2 of the SSL Baseline Requirements. We will post the incident report within the week.

Comment 2

[Ryan Sleevi]

Congrats on being the only CA so far to follow instructions :)

Comment 3

[Ben Wilson]

An update is needed.

Comment 4

[Arvid Vermote]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of this problem when we reviewed the issuing CA affected by https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 and internal discussions on remediation plans made it apparent that we would not be able to revoke all affected issuing CA within the timeline as set forth by the Baseline Requirements 4.9.1.2.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Time (UTC)	Activity
July 1 2020 21:06	OCSP EKU security issue disclosed on mozilla.dev.security.policy - refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1649937
July 7 2020 12:00	Second board & leadership debriefing on https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 and discussion on remediation activities for internal CA
July 7 2020 17:38	Final conclusion we will not be able to revoke all issuing CA affected by the OCSP EKU security within the timelines set forth by section #4.9.1.2 of the SSL Baseline Requirements and creation of this ticket

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

As detailed in https://bugzilla.mozilla.org/show_bug.cgi?id=1649937#c9 GlobalSign has ceased including the OCSP signing EKU in any newly generated issuing CA.

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Following issuing CA affected by the OCSP EKU security issue were not revoked within the timelines set fort by section #4.9.1.2 of the Baseline Requirements.

ID	CN	SHA256	crt.sh	Revocation date	Destruction date
1	AbbVie AATL ICA 2020	154E4834B28D4FB1F90FEE935D0DDE46C45A177FC1425A028C685C32855A85AD	https://crt.sh/?id=2369948023	October 21 2020	October 21 2020
2	CRB Group SMIME CA 2019	6A5F4C1678CA65E59F060D57CDFF665065314861D53A8E7D1450CA92D96CA102	https://crt.sh/?id=2029982659	October 21 2020	October 21 2020
3	DexKo Global SMIME CA 2019	ABC86706C98D6BF67372F908EC01ADF631B191D733AE89F8343EB047B108144B	https://crt.sh/?id=2029984306	November 18 2020	November 18 2020
4	GlobalSign AATL Partners CA 2019	83FC891B350D9E0D7EBE6DD2A6BFE3D0B0F4653FCA048615A5DEEBBC039A3F66	https://crt.sh/?id=1436918881	October 21 2020	October 21 2020
5	GlobalSign Issuing CA for AATL Partners 2019	67C46DC17762667844F1596089375FF45E05C2B316C89499F6E7FAB78C8F0379	https://crt.sh/?id=1703475173	September 16 2020	October 21 2020
6	GlobalSign Qualified Time Stamping CA 2019	74ABE5E5CCEB75491FF72C4CF325405D8ADBFE390E189CF430BA60E62798878E	https://crt.sh/?id=1490728721	N/A	July 28 2020
7	Qu\C3\A1litas Compa\C3\B1\C3\ADa de Seguros S.A. de C.V.	B716B089FE4E53D1A2EF7BA57AC85E68EC722CF61052C25A59626AD3B15C5F40	https://crt.sh/?id=1814826066	October 21 2020	October 21 2020
8	Ford Motor Company - Enterprise Issuing CA01	4C241CFE3D3FFB60CA88D6B06A552AB1CF0EF7D8D2E08DA15282B55192EBBD29	https://crt.sh/?id=392882654	Parent revoked on July 11 2020	N/A
9	Ford Motor Company - Enterprise Issuing CA01	3802E424516F78EEAC329AAE9B1F60A412DBE1D5B095D7AC9DC0DCDDE3C1F5FB	https://crt.sh/?id=306624237	Parent revoked on July 11 2020	N/A
10	GlobalSign CodeSigning CA - G3	4047C9D69260C07213BCB8608A7EC5E2838A56B79F67847812EAC0778D0D27F1	https://crt.sh/?id=157564305	April 21 2021	April 21 2021
11	GlobalSign PersonalSign 1 CA - G3	F068DEAA18CC02D5A8BE35CB8338327910291F6E62E7216A934764A1ABA4A800	https://crt.sh/?id=2369948051	July 28 2020	July 28 2020
12	GlobalSign PersonalSign 2 CA - G3	925EE7D5A22AD7FBE9BAB54D7C8D0B9A74F7E35A8AF6AF645E2E8C3519A7092F	https://crt.sh/?id=2369947954	July 28 2020	July 28 2020
13	GlobalSign PersonalSign 2 CA - SHA256 - G3	B778748A792B8F91F04B01BAFC31A31ED7EF6A712AFF80B6610D9AADEE207ADF	https://crt.sh/?id=24592899	February 24 2021	February 24 2021
14	GlobalSign PersonalSign 2 CA - SHA256 - G4	27D6FDAF80297846DFEFF82E7F58B9A48AC9E3EE93A112B1BBE243EE1A97447C	https://crt.sh/?id=2839140428	July 11 2020	July 28 2020
15	GlobalSign PersonalSign 2 ECC CA SHA 384 - G4	46038F6326228CDB56619C52266613DA04C8CA499E0D03B0EDCFFC110D5CFC70	https://crt.sh/?id=405618313	July 28 2020	July 28 2020
16	GlobalSign PersonalSign 2 RSA CA SHA 384 - G4	5CDD809CF44F5F8665EAC15055504C5B06B787AC18294505BDBAB4A77E50D776	https://crt.sh/?id=405618295	July 28 2020	July 28 2020
17	GlobalSign PersonalSign 3 CA - G3	B1FE3AEBF963A7880E74B0B0556681EA8B1CCCE3E69A7D3B10A68ACBE86E48A1	https://crt.sh/?id=2369948436	July 28 2020	July 28 2020
18	GlobalSign SMIME CA 2018	C8192C32F7B49C7F32A1CA001595A7F9E36C9E72058D6EAA1BAB7752A8C16718	https://crt.sh/?id=549505576	September 16 2020	October 21 2020
19	JCAN Public CA1 - G4	7B464DC384FDB1A525C2CC279ED0C7CFAD24BECF72C46A7D7093D157C217607E	https://crt.sh/?id=163676419	Parent revoked on July 11 2020	N/A
20	NAESB Issuing CA - SHA384 - G3	128DED1A8AD60C24B4254E31DB94FC4392BF93ED5434472AA43A0B9856106068	https://crt.sh/?id=2369948019	July 28 2020	October 21 2020
21	NAESB Issuing CA - SHA384 - G3	0986B5A1C7314EFB04FB648B9E2B57CF4842FD1D4345D28E52094C90A9FECBFE	https://crt.sh/?id=18068129	September 30 2020	October 21 2020
22	SHECA DV Secure Server CA	393B8B15CABC3886FB2E416495D63C8BADD8DCAF87552076C8A0A9637C24DE47	https://crt.sh/?id=1225556701	Parent revoked on July 11 2020	N/A
23	SHECA EV Secure Server CA	147C447FEEB86202B503314FCAF0036BEAAEF437C39B56B358EC446A9D20387F	https://crt.sh/?id=1229139434	Parent revoked on July 11 2020	N/A
24	SHECA OV Secure Server CA	77EAC476453CB732257FF166A5EBD1656CB1F673B68E28DF41774133979FA2A4	https://crt.sh/?id=1225556702	Parent revoked on July 11 2020	N/A
25	ATT Organization Validated CA 2019	7AA45D6F5B14DAB1C6844C19C2804E14B5811E6EDE1F02B0AEF065A7B359C68F	https://crt.sh/?id=1490728430	September 30 2020	October 21 2020
26	CrowdStrike OV SSL Issuing CA 2020	AE03B9AD17106A28785830B1DCD636797C4C64D81CB8D161595DBAF83433E64C	https://crt.sh/?id=2839140453	Parent revoked on July 11 2020	N/A
27	DPDHL Global TLS CA - I4	94C663E9EA5C27EE4F64127F9B425863E991A9E156C07DF1A00803AE31764162	https://crt.sh/?id=1814823951	September 30 2020	October 21 2020
28	DPDHL User CA I3	AF1898D7F0638751C075D0142D4E2A0EA731FC622324F153FE1BF3B6AFD9AF13	https://crt.sh/?id=1596016275	Parent revoked on July 11 2020	N/A
29	DPDHL User CA I3	2E0191751CA0CBA81C3A6338DEE1A02B8D6BCC4F1F8261B809BCCE7ABAF1A43D	https://crt.sh/?id=12729527	Parent revoked on July 11 2020	N/A
30	DPDHL User CA I3	BCE3A5BD8D9082636C5BFE3E0B71ACEE551E24E3BD035887D2661ADA65AFF484	Not in crt.sh	Parent revoked on July 11 2020	N/A
31	DPDHL User CA I3	F037621405E0F356507E239FADD647842D3B50857C3CFF840859174F72F6FD18	https://crt.sh/?id=329514052	Parent revoked on July 11 2020	N/A
32	DPDHL User CA I4	C25C4EDBC36E3FB7C3D937BEE9F2D29E36AFB07CFA3188262E0D5FDC919E0D77	https://crt.sh/?id=2369948075	November 18 2020	November 18 2020
33	Giesecke and Devrient CA	632FD697BACAF1ED232517EC9B7622B7C25E1448B0CC626B33286719E351CE8A	https://crt.sh/?id=196919504	Parent revoked on July 11 2020	N/A
34	GlobalSign CA 4 for AATL	EBA34C7B109671614C367E1DE075124C3954CE19F85FACF61090EC319F7F1A7F	https://crt.sh/?id=1229139435	December 31 2020	January 20 2021
36	GlobalSign CA 5 for AATL	306E9739E3458FF4546877B704B2E3905E58B235D64E32F4F026AC91B7295D15	https://crt.sh/?id=408789250	December 31 2020	January 20 2021
37	GlobalSign CA 6 for AATL	BE1FFC0E1FF6088104F43E327E7C7DC72A9CA7B0DF05793123ABE32DEACEE76F	https://crt.sh/?id=2369988390	December 31 2020	January 20 2021
38	GlobalSign CA for AATL - SHA256 - G3	2E8820DC0EAFAE3D6D285C057ECE14470B377438B002CEDD4C72B4F343A54F43	https://crt.sh/?id=2369947889	N/A	January 20 2021
39	GlobalSign CodeSigning CA - SHA256 - G2	BE40813869AB27A071D12AD6A8830583EBC3B618E3F2346359F4B11A1C9434EE	https://crt.sh/?id=1703475054	October 21 2020	October 21 2020
40	GlobalSign CodeSigning CA - SHA256 - G3	FB54EEA9BCE8E9EA9782154F3D414277FB709F49B947D73978AC278546C2CE03	https://crt.sh/?id=26749929	April 21 2021	April 21 2021
42	GlobalSign Extended Validation CodeSigning CA - SHA256 - G2	1E864278C20881B671C0C6D2E14B61150AD1F13CF92C6EC14B550DCBC47E1541	https://crt.sh/?id=1703475088	October 21 2020	October 21 2020
43	GlobalSign Extended Validation CodeSigning CA - SHA256 - G3	DD038E87E0B4D2C369680D3DE78638AB39FC1D7E50632996921101768DB8D4D8	https://crt.sh/?id=41285443	April 21 2021	April 21 2021
45	GlobalSign HV RSA DV SSL CA 2018	54C37A8E853FD1D6378D378B939307EC321A31CC1A5A89E7180633BC13F18762	https://crt.sh/?id=970082980	October 21 2020	October 21 2020
46	GlobalSign PersonalSign 1 CA - SHA256 - G3	F5D2D2BA6817A7A9AA0E21354BBF0E6F95C5E287EE88CF2F279F0FFEC4EDAC15	https://crt.sh/?id=147619379	February 24 2021	February 24 2021
47	GlobalSign PersonalSign 3 CA - SHA256 - G3	701B432AC0CDD4D9CF95B4B884C32BF5CCA90D44E0161ABD13B934D68E380472	https://crt.sh/?id=163079175	February 24 2021	February 24 2021
48	GlobalSign PersonalSign Partners CA - SHA256 - G2	4E707867946AC05343C6BA8FF121EA66A758037913257A8EE4974350D39A1034	https://crt.sh/?id=2369948041	January 20 2021	January 20 2021
50	GlobalSign PersonalSign Partners CA - SHA256 - G2	118262C2088EE1528E20D836D2070854707C0D8F8E80FBE396F9ECD4B9141B5B	https://crt.sh/?id=12715740	September 30 2020	January 20 2021
51	GlobalSign Qualified CA 1	F5709A2D2F68B53BF6F645BB178ADF95346F89FDA5C63BFDE08042A26492AAB2	https://crt.sh/?id=509714293	February 24 2021	February 24 2021
57	GlobalSign RSA DV SSL CA 2018	9E898ED03FA46969690DAD73C7296675045FF9B5A0100A399BEB8435A98F5185	https://crt.sh/?id=970083106	January 20 2021	January 20 2021
58	GlobalSign RSA EV QWAC CA 2019	EDC734C501501DC7A27448FA02C74931F8578BF297B173F34B841E82C6691926	https://crt.sh/?id=1490728500	February 24 2021	February 24 2021
60	GlobalSign Timestamping CA - G3	95C6A747DD0BC755A1941827E894B8083592241B792541E2EB1B30FB9B13F57F	https://crt.sh/?id=2392141070	N/A	May 28 2020
61	GlobalSign Timestamping CA - SHA256 - G3	BE33D1C57EBDDD927B57BDB604BE457B552FE568E7F3DCBA093C39ED1C30A239	https://crt.sh/?id=2369948437	N/A	May 28 2020
62	LinQuest SMIME CA 2020	113138DD7B216725840238E2D7EEECB3738DB139064B24CB853FC270A49E6057	https://crt.sh/?id=2369948433	November 18 2020	November 18 2020
63	NAESB Issuing CA - SHA384 - G4	C4C7C436BD88E8E68DB00297DF83ACC819E198639BA00522C8E3245876898523	https://crt.sh/?id=2369948432	February 24 2021	February 24 2021
64	RNP ICPEdu OV SSL CA 2019	42CFDDA6F660B8E5B4C1C411965A4519312559E3262F8DB69D2DAE17B26B3BA3	https://crt.sh/?id=1476651440	Parent revoked on July 11 2020	N/A
65	Trafigura PTE Ltd S/MIME ICA 2020	5E7FCB9C97BDA56993B1658D120232761D665A3644534300FA6A5BEC5E0D5795	https://crt.sh/?id=2369948428	July 11 2020	July 28 2020
66	GlobalSign CodeSigning CA - G2	FFFE077503FD72F0E5338B0A7B4E218E7D1FF82E493E7E852AE51AA1C7585D17	https://crt.sh/?id=1476651569	October 21 2020	October 21 2020
67	DPDHL User CA I3	EBE87BB4188502709F444055259ABB22BC51B88C908419A13559DFC8EF6630D1	https://crt.sh/?id=12729526	Parent revoked on July 11 2020	N/A
68	Ford Motor Company - Enterprise Issuing CA01	CF73B52D041B7309B439D16247414B90C9D26E44E38748A36500D5829B5187F9	https://crt.sh/?id=215376217	Parent revoked on July 11 2020	N/A
69	Ford Motor Company - Enterprise Issuing CA01	3B9668F59F55FA3838FC2A3B80B7F9B5B13D1A46F1EAA6E0BCFF04C54198056C	https://crt.sh/?id=215376215	Parent revoked on July 11 2020	N/A
75	Liberty University External Issuing CA 01	1F91212C6BFC333C6EB52A685525E1E5B9E3AC1EF7A5A86649F5F95C721D8898	Not in crt.sh	Parent revoked on July 11 2020	N/A
76	Liberty University External Issuing CA 01	CA005AA75E33594BD1DEDC584E1E74E5198EBB1DE88929ED4F3E2E9FFCE3873B	https://crt.sh/?id=36391364	Parent revoked on July 11 2020	N/A
79	Crown Prince Court CA	BF5EDFBEEB85999C5169CBF3F4DB63B679AD2E1E2272FC3795F9F9921E6D0487	https://crt.sh/?id=7890405	Parent revoked on July 11 2020	N/A
80	Crown Prince Court CA	A0133BE5B14E02310A2D4BEAB601094F1194EE8BD6FD29DDFE7B9347467C2EEC	https://crt.sh/?id=10105729	Parent revoked on July 11 2020	N/A
81	Crown Prince Court CA	F164AD5E4CE9EFC0A144CA902EA2ED46C464D2D508CA919A23095CDF30D4DC68	Not in crt.sh	Parent revoked on July 11 2020	N/A
82	Crown Prince Court CA for AATL	DF45EEAED905C58D730EC5497B59B3AB4CCE7C6459953DF9CA5C1F031AC06DD8	Not in crt.sh	Parent revoked on July 11 2020	N/A
83	Crown Prince Court CA for AATL	D21076207F79A9B04137D40A4FFE6DD08921CCF49E6EB60277FF4593E076D538	Not in crt.sh	Parent revoked on July 11 2020	N/A
84	Crown Prince Court CA for AATL	70BDB19C31F5EF105B29376E35EA5ED8EEBE13CB5C0758C32DFC4C5F7230A173	Not in crt.sh	Parent revoked on July 11 2020	N/A
85	DPDHL TLS CT CA I3	9153E4420DDC7EB4E6E864AA0377DADF4082ECD35052113638E05D3C296BC006	https://crt.sh/?id=786990298	Parent revoked on July 11 2020	N/A
86	DPDHL TLS SHA 2 CA I3	25BACC40A5392B82AADEA04903905A467121F28220E6F2F7E0FE982AAFC14FA6	https://crt.sh/?id=8527797	Parent revoked on July 11 2020	N/A
87	DPDHL TLS SHA 2 CA I3	5A405535C112A0A81AF0D2ACCA3C3F9BC1A677586CDBC633CB4F5F778E1A3550	https://crt.sh/?id=329514048	Parent revoked on July 11 2020	N/A
88	DPDHL TLS SHA 2 CA I3	23A74704D77A03CFD3FF19E62C500848214E6C60FD2AAEF7DCE7A8F9EE9F9232	https://crt.sh/?id=27823827	Parent revoked on July 11 2020	N/A
89	DPDHL TLS SHA 2 CA I3	1C942A22A016A1E5559DAE77EC5CE8671F98AE0BA4AC2DC259418E8E1E9F94AD	https://crt.sh/?id=9881176	Parent revoked on July 11 2020	N/A
90	Southern Company External Issuing CA 1	FB953C4FC0045846D02491C8ECCF387BA34347C17ABB0EA6D59F6DE4D2F1EA04	https://crt.sh/?id=11501550	Parent revoked on July 11 2020	N/A
91	Trusted Root CA G2	6E32A35B599E9087BB1AB35CE73022EC2E26AF34BE388919419C95700CD8E7FB	https://crt.sh/?id=1862521	July 11 2020	N/A
92	Trusted Root CA SHA256 G2	01FD73EF5E70F526FC9C11F65FE2EE6F7125B3693949227FFD8E459E583C458A	https://crt.sh/?id=3179271	July 11 2020	N/A
93	CBMM SMIME CA 2019	C346D9137E05254C6EEAC99AC2F6748A0C5D3AFC6B7B9B1E00C40ADF4D85655D	https://crt.sh/?id=1596016282	November 18 2020	November 18 2020
94	Accenture Federal Services External CA	A9C8E971259A2ED6E65F721E07AA967C72C7CDB47C7BE1288D87BF08D2F3580D	https://crt.sh/?id=215376216	December 31 2020	January 20 2021
95	EY LLP SHA256 CA - G2	1557F65BA61C958B74EFA4A582BBAEBDD62A6D9B65FE95A80D5ED518F46ED87F	https://crt.sh/?id=970084237	December 31 2020	January 20 2021
96	GROB-WERKE GmbH und Co. KG SMIME CA	1608BF87414CDCFAB4279102A19702D9D5996A91329E3DF2F80495473AAD86C6	https://crt.sh/?id=872293772	October 21 2020	October 21 2020
97	Hyperion SMIME CA 2018	DCD77E34B46D530AEF645A513389CD4FFC0F7D196A115B8F62A5FD0D557D46C6	https://crt.sh/?id=721305503	November 18 2020	November 18 2020
98	Spirit AeroSystems SHA256 CA - G2	8609BDCEF95E4A4D426497B5CD8ED4B001C953A5C14471CAAF58FB650DF8ABF0	https://crt.sh/?id=215376218	November 18 2020	November 18 2020
99	HERTZ SHA256 CA - G2	64FD7F66B805EF0FAE09DAC06EAD1A9AB5C28E6F24AD0759996D349987FEE7E2	https://crt.sh/?id=408789253	October 21 2020	October 21 2020

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Refer to the table above.

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

1. Moving non-TLS use cases away from TLS roots

A significant portion of the affected CAs relate to non-TLS use cases: document signing, time stamping, S/MIME, authentication, code signing and esoteric PKI ecosystems such as NAESB, eIDAS and AATL.

As detailed in https://bugzilla.mozilla.org/show_bug.cgi?id=1599788#c4 GlobalSign created a new set of roots March 2020 with the aim to separates different PKI certificate use cases under different roots - the naming scheme (R|E)[0-9]{2} is R|E for the algorithm, [0-9]{2} for the year of root expiry.:

• GlobalSign Secure Mail Root R45
• GlobalSign Secure Mail Root E45
• GlobalSign Code Signing Root R45
• GlobalSign Code Signing Root E45
• GlobalSign Document Signing Root R45
• GlobalSign Document Signing Root E45
• GlobalSign IoT Root R60
• GlobalSign IoT Root E60
• GlobalSign Timestamping Root R45
• GlobalSign Client Authentication Root R45
• GlobalSign Client Authentication Root E45
• GlobalSign Root R46 (TLS)
• GlobalSign Root E46 (TLS)

The above roots were included in our latest WebTrust audits dated June 29 2020. Since the delivery of the WebTrust reports and inclusion in our CPS we have started enrolling the above roots in the relevant root programs. The initial plan was, once sufficient trust and ubiquity within the context of the use case was obtained, to generate future issuing CA and leaf certificates under these new roots as of 2021 with as endgame having fully seperated hierarchies, in order to reduce the scope, WebPKI impact and remediation effort of any future similar compliance incidents.

In the context of remediating the EKU OCSP incident we are accelerating this effort: where obtaining the necessary ubiquity for their use case takes time, we are cross-signing the roots with current generation roots and for those use cases where issuing CA enrollment into an esoteric PKI ecosystem is required (e.g. eIDAS, NAESB and AATL) we have started those activities. Affected issuing CA with a non-TLS use case will be migrated off our current generation of mixed use case roots to dedicated roots, after which we will revoke the affected issuing CA. As of now, future issuing CA with a non-TLS use case will be generated under the next generation of roots (under the condition this does not cause any trust or ubiquity issues).

2. Key management bandwidth

In the context of business continuity and responding to CA key compromise we have long maintained the capability to perform revocation key ceremonies from three different key management locations within a short amount of time (24-72 hours depending on the location). At the time of the OCSP EKU incident only one of these locations had the capability and resources for generating new issuing CA keys and certificates with no lead time (we have a secondary location with that capability but the lead time to perform the actions there is a few days). Given the amount of issuing CA affected and corresponding key management tasks in terms of re-generation, revocation and destruction it became apparent we need to expand our key management bandwidth in order to perform these different key management actions more timely in the future.

We have commenced the activities to render a second key management facility and associated resources capable of performing expand key management tasks with no lead time and ensuring more available bandwidth for future incident response activities. This is expected to be completed November 16 2020.

3. Increased rotation of TLS issuing CA

For our new platform "Atlas" we are currently architecting and building more frequent rotation of TLS ICA so the amount of leafs that chains to them is contained, which will allow a more timely re-issuance of leaf certificates and revocation of their affected issuing CA.

In 2021 we will rotate the GlobalSign TLS issuing CA on the Atlas platform every 6 months and customer-dedicated TLS issuing CA on the Atlas platform every year, further increasing the frequency of GlobalSign TLS issuing CA rotation to every quarter in 2022. Our legacy platforms are expected to be fully deprecated for TLS in 2023.

4. Automation

GlobalSign has been actively promoting APIs for customers to manage their certificates and we are continuing to build on those to expand automation and provisioning.

1. We're in the final development of our ACME services which will enable fully automated domain validation, issuance and provisioning of DV and OV TLS certificates for service providers and Enterprise customers. This will be available to select customers in Q3 of this year and to a wider audience in 2021 as more customers use our Atlas platform.
2. Our Automated Enrollment Gateway, AEG, is an on-premise proxy solution which automatically provisions TLS certificates based on Active Directory policies using auto-enrollment. AEG also supports non-domain joined platforms via the use of ACME for certificate issuance. AEG permits enterprise customers to centrally manage their endpoints and to initiate and push out new certificates without needing to initiate the issuance or replacement from the web server.

Comment 5

[Ryan Sleevi]

Some of these CAs stand out as quite long, e.g. "GlobalSign CodeSigning CA - G3", "GlobalSign PersonalSign 2 CA - SHA256 - G3", "NAESB Issuing CA - SHA384 - G4", or even "EY LLP SHA256 CA - G2"

What controls are in place to mitigate risk, and what's GlobalSign doing to provide assurance in the interim of the correctness of those controls and their assessment, e.g. such as the provision of a Detailed Controls Report?

Comment 6

[Matthias]

Time (UTC)	Activity
July 1 2020 21:06	OCSP EKU security issue disclosed on mozilla.dev.security.policy - refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1649937
July 7 2020 12:00	Second board & leadership debriefing on https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 and discussion on remediation activities for internal CA
July 7 2020 17:38	Final conclusion we will not be able to revoke all issuing CA affected by the OCSP EKU security within the timelines set forth by section #4.9.1.2 of the SSL Baseline Requirements and creation of this ticket

Could you update your timeline to include all relevant events leading to the delayed revocation? The current published timeline raises the concern that GlobalSign, even with the mentioned remediations, will not be able to revoke their intermediate CAs in time because the first related activity after the problem report is a debriefing at the day before the BR deadline.

Comment 7

[Arvid Vermote]

(In reply to Matthias from comment #6)

Time (UTC)	Activity
July 1 2020 21:06	OCSP EKU security issue disclosed on mozilla.dev.security.policy - refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1649937
July 7 2020 12:00	Second board & leadership debriefing on https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 and discussion on remediation activities for internal CA
July 7 2020 17:38	Final conclusion we will not be able to revoke all issuing CA affected by the OCSP EKU security within the timelines set forth by section #4.9.1.2 of the SSL Baseline Requirements and creation of this ticket

Could you update your timeline to include all relevant events leading to the delayed revocation? The current published timeline raises the concern that GlobalSign, even with the mentioned remediations, will not be able to revoke their intermediate CAs in time because the first related activity after the problem report is a debriefing at the day before the BR deadline.

The supporting overview of events is available in https://bugzilla.mozilla.org/show_bug.cgi?id=1649937#c9.

Time (UTC)	Activity
July 1 2020 21:06	Security issue disclosed on mozilla.dev.security.policy
July 1 2020 21:42	CISO is notified of the post and the fact GlobalSign is affected
July 1 2020 22:22	CISO notifies the leadership and compliance team and mobilizes an investigation team
July 2 2020 19:31	Investigation and impact analysis completed, initial remediation plan finalized: focus on remediation of hierarchies containing third-party operated issuing CA keys first and remediate the affected issuing CA / keys under GlobalSign control as a second priority
July 3 2020 01:00	Key manager and compliance team start preparing for initial re-issuance of issuing CA within hierarchies that contain third-party operated issuing CA keys outside of GlobalSign's controls (Trusted Root)
July 3 2020 01:00	Incident team starts preparation activities to work with customers affected by the initial revocation activities
July 3 2020 13:00	GlobalSign board & leadership are debriefed by CISO and approve the plan of action: revoke the affected Trusted Root hierarchies (the GlobalSign hierarchies that contain third-party operated issuing CA keys outside of GlobalSign's controls) on July 8 16:00 UTC
July 3 2020 14:00	Incident team starts to work with customers affected by the initial revocation activities to re-issue of certificates under alternate hierarchies or prepare for swapping their issuing CA with a new one to be generated on July 5 2020
July 5 2020 09:00	Key ceremony to generate new issuing CA for affected customers under Trusted Root hierarchies
July 5 2020 11:00	Key ceremony concluded, start setting up new issuing CA and re-issuance activities for customers
July 7 2020 12:00	Second board & leadership debriefing and discussion on remediation activities for internal CA
July 7 2020 13:00	GlobalSign receives official letter from a regional infrastructure provider in Japan, which has leafs under the "JCAN Public CA1 - G4" (https://crt.sh/?caid=51044), chained to the "Trusted Root CA SHA256 G2" (https://crt.sh/?caid=1423). The letter detailed that a revocation of the hierarchy on July 8 2020 UTC would have significant impact on the companies' ongoing relief and recovery efforts related to restoring their services in the context of the ongoing flooding disaster on southern part of Japan.

Comment 8

[Arvid Vermote]

(In reply to Ryan Sleevi from comment #5)

Some of these CAs stand out as quite long, e.g. "GlobalSign CodeSigning CA - G3", "GlobalSign PersonalSign 2 CA - SHA256 - G3", "NAESB Issuing CA - SHA384 - G4", or even "EY LLP SHA256 CA - G2"

What controls are in place to mitigate risk, and what's GlobalSign doing to provide assurance in the interim of the correctness of those controls and their assessment, e.g. such as the provision of a Detailed Controls Report?

We are currently discussing with a qualified auditor on the options to provide an ISAE3000 SOC2 Type I/II report on the non-performance of OCSP signing by and protections in place for the affected issuing CA & keys. We'll share an update once we have made progress on the matter.

Comment 9

[Ryan Sleevi]

Checking in for an update/progress on this point? I'm not sure a Type I report would be that useful, but perhaps you can share more details?

It seems a WebTrust Detailed Controls Report would be more relevant?

Comment 10

[Arvid Vermote]

We thought of reporting on a customized set of controls including relevant controls from existing requirement sets (e.g. WebTrust Criterion 6.8) but also tailored controls not directly included in any of the requirement sets but relevant to the context of this incident and associated risk, to make sure the following information (and other system design features) as reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 are sufficiently covered - please let us know if this is not the expectation:

Knowing that this does not provide any assurance, here are some additional insights into our environment:

• Even if they were intended to be potentially used for that purpose in the long term (OCSP response pre-generation directly from the CA), none of the affected issuing CA and associated keys have effectively been used to sign OCSP responses, we use dedicated OCSP responder systems, certificates, keys and HSMs for that purpose.
• Our CA issuance systems are logically set up in such a way that only outbound push/pull queuing from the CA issuance segment to another network segment is possible. No other interactions between the CA issuance segment and other networks are possible.
• CA issuance systems are stripped of any unnecessary functionality and do not contain the necessary software to generate OCSP responses (short of manual generation using pkcs11-tool or openssl, which would be captured in the audit logs)
• Apart from logical segmentation our CA issuance systems and HSMs are physically in a separate zone (room) with multi-person access requirements, preventing the CA keys to be exposed or exported to other systems such as OCSP responders.
• We maintain configuration records and system change logs for all systems involved in CA operations
• Each key signing action on CA issuance systems is logged

We would also prefer to share the draft system design description and list of controls with the community first prior to starting the audit activities to make the reported matter will match the expectations of relying parties. Given current progress we expect a first draft to be ready by August 21.

I also thought that providing a Type I (to provide assurance on the design of the system) as soon as possible, and a Type II (covering the period from the security risk having been raised to the issue being remediated) would provide additional assurance compared to accepting our claim that we do not use ICA keys for OCSP signing, and only receiving an assurance report after a considerable amount of time (i.e. when the issue is remediated).

Comment 11

[Arvid Vermote]

Update - we are still working on the system design description and list of controls. In the meanwhile any suggestions, expecations or feedback on the reporting approach are appreciated.

Comment 12

[Ryan Sleevi]

Arvid: Do you have an update here? Comment #10 mentioned August 21, and while Comment #11 noted a delay, I haven't heard an update.

Comment 13

[Arvid Vermote]

Attachment: GlobalSign OCSP EKU incident SOC2 control proposal v1.pdf

Please find attached a first proposal of the controls and testing activities to be covered in the custom SOC2 report on the OCSP EKU incident. The idea is to obtain a SOC2 Type II with a period-under-audit covering April 22 2020 through April 21 2021 (the date on which the last keys associated with CA that have the OCSP EKU included will be destroyed).

We have selected the controls which we deem most applicable in the context of the incident and associated risk or where a more specific population (i.e. a random sample or full population of OCSP EKU-containing CA) compared to the regular WebTrust audit makes sense. We have already engaged with a qualified auditor for the execution of this work.

Any feedback on the proposed controls and tests or suggestions for control expansion to provide more assurance are welcomed. We are also in an ongoing discussion with the auditor whether we can also have a SOC3 version of the same report to share with any relying party.

Comment 14

[Ryan Sleevi]

Arvid: Thanks for attaching this, and apologies for the delay here.

With respect to Control 2.1 of WebTrust for CAs, the wording as part of the test includes the following language:

based on risk assessment and in accordance with the CP and CPS.

This seems to be at odds with the Baseline Requirements, Section 6.2.7, which does not include a risk assessment with respect to protection of CA key material, as it uncategorically expects them to be maintained in such devices. This seems like it can be corrected with regards to the test to be applied.

Control 3.3 is perhaps the most important and relevant to the community here, and I'm somewhat concerned that the test to be applied doesn't describe the set of tests the auditor(s) will be performing in order to reach the necessary assurance level. We'e seen from past incidents some confusion and misunderstanding, and while I'm thrilled to see a GlobalSign internal control explicitly included, I'm also concerned that without understanding the procedures, it may provide limited assurance to relying parties. For example, one could imagine an auditor (incorrectly) examining certificate and CRL issuance databases, and only seeing certificates and CRLs, rather than examining the HSM activity logs and correlating those to said databases and manual ceremonies to ensure no unattributed use of the key material. I think this similarly touches on the tests for Control 7.1

However, I do want to commend GlobalSign for again, by leaps and bounds, providing an exemplary demonstration of the transparency and detail that we expect of all CAs. Comments like Comment #13 are the model for what CAs should be achieving here.

Comment 15

[Arvid Vermote]

We are currently finalizing the SOC2 controls incorporating the feedback provided in Comment #14, and will share the final version as soon as we've reached consensus with our auditor. The following remedial action related to key management bandwidth has been completed on November 9 2020.

2. Key management bandwidth

In the context of business continuity and responding to CA key compromise we have long maintained the capability to perform revocation key ceremonies from three different key management locations within a short amount of time (24-72 hours depending on the location). At the time of the OCSP EKU incident only one of these locations had the capability and resources for generating new issuing CA keys and certificates with no lead time (we have a secondary location with that capability but the lead time to perform the actions there is a few days). Given the amount of issuing CA affected and corresponding key management tasks in terms of re-generation, revocation and destruction it became apparent we need to expand our key management bandwidth in order to perform these different key management actions more timely in the future.

We have commenced the activities to render a second key management facility and associated resources capable of performing expand key management tasks with no lead time and ensuring more available bandwidth for future incident response activities. This is expected to be completed November 16 2020.

Comment 16

[Ben Wilson]

Please provide a status update on these ICAs. Thanks.

Comment 17

[Arvid Vermote]

Hi Ben - please refer to https://bugzilla.mozilla.org/show_bug.cgi?id=1649937#c30, the next batch of CAs will be revoked on January 20 2021.

As for following remedial action:

3. Increased rotation of TLS issuing CA

For our new platform "Atlas" we are currently architecting and building more frequent rotation of TLS ICA so the amount of leafs that chains to them is contained, which will allow a more timely re-issuance of leaf certificates and revocation of their affected issuing CA.

In 2021 we will rotate the GlobalSign TLS issuing CA on the Atlas platform every 6 months and customer-dedicated TLS issuing CA on the Atlas platform every year, further increasing the frequency of GlobalSign TLS issuing CA rotation to every quarter in 2022. Our legacy platforms are expected to be fully deprecated for TLS in 2023.

We have generated the first set of CA that will follow this rotational scheme:

• GlobalSign Atlas R3 DV TLS CA 2020-12
• GlobalSign Atlas R3 OV TLS CA 2020-12
• GlobalSign Atlas ECCR5 DV TLS CA 2020-12
• GlobalSign Atlas ECCR5 OV TLS CA 2020-12

They are currently being set-up in the Atlas platform and we will swap issuance to these new CA at the end of this month. The next swap will be in July 2021 and as of January 2022 we will swap TLS issuance every quarter.

Comment 18

[Arvid Vermote]

Hi Ben - I propose we close this bug and continue the tracking of the SOC2/SOC3 report on non-performance of OCSP EKU signing in https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 as that is the one our customers and other relying parties are watching in the context of this incident. Thank you.

Comment 19

[Ben Wilson]

I will close this, and we can track any additional remediation action in Bug #1649937.