1413761 - DigiCert / Symantec: EV JOI Issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36

Steps to reproduce:

I wanted to report the following mis-issuance of EV certificates. I didn't find a report from Symantec from back when this happened, but I apologize if it's duplicating another bug. Note this only applies to the systems we acquired from Symantec, not DigiCert's (which means the system supporting the Sub CA plan is not at issue). I thought I'd post all of the issues we discover that weren't already closed to keep things transparent, even if older.  The information is pretty old, but I'll have the team dig up any addition info required.

1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
On July 20, 2017, a validation analyst identified and reported an existing EV certificate with incorrect Jurisdiction of Incorporation (JOI) fields. Other certificates were discovered through addition investigations. 

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
a.	Certificate discovered with incorrect JOI on 2017-07-20
b.	Investigation begins at 2017-07-21 to determine if the one occurrence reflected a systemic problem or a personnel training deficit. Initial review suggested issued was not reproducible. Case was left open.
c.	Follow up on the case was escalated 2017-10-04 19:39 PDT
d.	Cause was identified and verified 2017-10-05 04:45 PDT 
e.	Root cause analysis started 2017-10-05 11:07 PDT
f.	Software patches were applied to our system on 2017-10-05 to correct the source of JOI information used, 2017-10-10 to address an edge case detected after 2017-10-05, and 2017-10-25 to introduce an additional safeguard by forcing certificates into manual review that otherwise would be issued via automation through reliance on data reuse rules.
g.	Initial extract of candidate certificates from our database on 2017-10-05 to locate certificates with similar mis-issuance. We further refined extracts through 2017-10-13.
h.	Analysis of all candidates completed 2017-10-16 16:45 PDT based on extracts occurring between 2017-10-05 and 2017-10-13.
i.	Revocation of impacted certificates occurred within 24 hours of problem confirmation in each specific certificate and occurred between 2017-10-10 and 2017-10-17 16:45 PDT

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Symantec stopped issuing certificates with the problem as of 2017-10-10.

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Symantec found a total of 387 EV certificates with incorrect JOI fields. The first was issued 2013-03-11, and the last was issued 2017-10-10. The 108 still valid certificates were revoked. The remaining 279 certs had already been revoked or were expired.

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

They are EV certs so all of them should be in CT logs. The serial numbers are:
0b87cfc6300ac5e5e413bdd7fda91b49
79b46cf0e2b12bbe24cdf8191ea0fd1c
1a3b02cb7d69e6b9473ad85448e910ee
6ffab89edbf1c4fef8744524f96f8f7b
07f3f45a72f682989c4343fced576a7c
678852d5cc1ff1948cb1ab12cdf71249
51db3c9040f3911f6ec7d0c0e3dd8816
053e8cbd0e813907aef1d08a5d06c6b7
406874efb9ceb7d39a0de3a35e30e111
6de8a6771acd5554a5ca160101c13114
1f873ebfe850dfa153211cd928cfb565
3351836924dcd3026bb534b0bc17ef27
3666b4fc117f873d3b836722a7fea49f
7a163097a6bb2f750057583afa1b6d0a
2f8c502085c13127455ba4c5e34c0b91
3965928dae4f4c00e4d56815479a0722
20be9c85b6599c5ceb4db5ff5d7b946a
7a8ee91821eda10c429fd738f6e83202
055083056c525d14a9914e7400cd8bc3
7b9f0a59b1ca08dffdb2f89dfbaabfb8
4171f77eeebd6b0b53f8eba9361c1ef5
7a648d6d4bebe9cf9361c6833f78dfbd
46891614f5dc87092a4221b6df2b157a
395d1b5bf3f709b47056828a5463dd7d
03ceb22e0ed816d5284414c8b37c7179
5cd51da9d39570fcd2fe6a020a8c6171
7f9db327e5b236391f31da61477a59b8
46c8a7c8ce35116d9a06de4dca6e5380
5fe9d19c435fb89c0d753c6d26d19b93
0cd7dc8cfaadd902bee81125455dd636
16344550954324baa78d3392efdb83c7
5986b2f01248f7cd7875a535e6a2f9d5
0e8550b6ff1f12dcf1a560978275f371
533852556e1aae93a2ca063631aa499c
4c081134a842ad09f85ea90d0270751e
6f6aad895eb65c8fbe12c1990e6ced76
18b144319290b9d7b0e519fd0ccc4331
62c864954844965ca2f433431fb568c9
27617b1e86a51200f53529cc2731cb52
58336f7f8a25fb18222e75cb431389a1
1a39d61e38336eaad8435c2f605aa987
027f7dacf035aa80522ae7ba80fdb5b0
342ed8fcab5f472b481b8e565e5cf769
117a4e73f669bd3b0b145581087bb4f0
1b7d71985d2480d533761f87cb274b10
15a93838a97ad113b39d434f5cc5be29
485e8b5d2ebbb4ab963e0605ed949633
418c32860b574a13cd9c6862e38e134d
6fcc37f2403aa23e835b420a25df5409
7f70ef9e29489e185737458f6c38d2b2
472f317a9dbe53b557e5e6633e707b5b
7da921092b24d06c8fac19c4047e4c75
1d838eb488d57b2c9c8d9d1a7614c7e3
4d77a5e5a3f877aac74a333b7c82ebf9
0c0122b7dc928cdf11bde7c120a5aa80
302a98dc1d474405c9c082652e11bd61
1c1f74ea320bce810a16c96522675e45
1a7863eb14faf2d8d406fbbcb6239e3a
792040f434cb3dbfd4012e974120a944
01eeb9b156c060d50ca5cab7ddf5c8fe
4907fc19c6910359e150252a849319cc
6dd3e48c0be6994171e911da18e77f05
13ffa1e3c075584500efe42a4f9206be
3c83757d2db3d2f7c313d7c6f4b3dbf1
6f92781253cf0811807fa53189e0116a
3c8000f9715da94d01a9c2bebe24b156
5ea8df1f60c618313670cd6cd40a649d
69d83063680cf6142c41d538009c7aa2
555f1d3d6b98312cddc4018562c25450
582f53a13f2b5edfb4051dfdf7ae0c79
22d2dce5136d3fe1abb4c6b61e039d63
41778e580fec2bbd74dc067cc59ac962
3985165aa81ade7294b365e81b1e7721
73323e6344764ffecec9b7de517275ab
670d27a72a5fc599b90f98f1fb7854c0
0c57d7db2775a5b932b81a5704ea0676
3eef7434aa529e66992258a9fdd4f09e
1d90223e067adf6b0e18b94012b5c007
118ea8a4be4154c77d46018a3cb0efc6
4634980dfcb9bf2fd4e67ef2e260d979
054ee3b4b26a82ccadb4e802a524d38f
6119337e9b8e68a9ad3c6b0879f9f6f2
2b9350c978f63408da32dd7d9020c263
06073187074464edc6848e3abf0e154d
5fc246861883780e7ee78790573e7483
5fbcf64750a7ed75f76255398092ccb1
27e21bf630866be5cdb21dc1c9cdaa55
7bd81b30f5e22d69f23753a592458c7d
36f846397abe96071255772687b65fa3
152350a23448fb13742c42f35844ac57
0f52b7c9c0c15b0bc5a96c102d02d251
22e9ff427e944f4ec0266e0a61091dd9
1efa3036a59ab0088cea6367e32b023d
0d337ec2e534836de1a8b8c02d3ff1cf
349d716bef139d13b95391069fd4b26a
22d8ce5074a5488bf290a572421d71ef
4ae3f487c65b05915f52a0bba77c4dd9
29fbf0608108132732adf1d23010e975
705be4e68381875ba4df49ef13be77fa
1b73fd72e0c3f83d02ba7988d0ba04c5
09ea44ee9f94ca1651d58af16d2a4ed4
124b531ebe32a41ead6ce797ec8c293c
671fe934eaaeec44fde9f073695edd81
5002325b0a4501c3db858ece96bf6a11
76ef41cee360a25d5d9b6c4e2c773ae1
7eb40186398e3086cb91fd154d1d37dd
5fb9313ae1eb543585714110f6d066fe
5e05f8b9a4fcfc2ce3b111aeffe42a94

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Symantec offered certificates containing, and signed with, RSA, ECDSA and DSA keys. Some customers received “sibling” certificates, in which the customer would purchase a certificate with one type of key and then request one or two siblings with keys from different PKI algorithms. The intent was that the sibling certificates would contain different keys yet identical Distinguished Name information, but the team discovered that in some cases, sibling EV certificates contained incorrect JOI fields.

When Symantec received an order for an EV certificate, the cert was saved in the appropriate database, and pre-populated the JOI fields in the database from the Country, State and Locality of the Corporate Contact, aka Certificate Approver. During the normal course of validation in some cases as the team investigated the EV applicant, they determined that pre-populated JOI data was incorrect, they corrected this information, and saved it into the database. The dev team discovered a bug in its code in which the validated JOI fields were included in the original EV certificate, but the siblings contained JOI fields based on the initial Country, State and Locality of the Certificate Approver. 

The bug was introduced due to an oversight by a developer who failed to capture the case where initial placeholder JOI information could change during EV certificate validation requiring that the sibling be populated by the validated result rather than the initial placeholder data. The system also lacked an automated test case for JOI fields that differed from the Country, State and Locality of the Corporate Contact. 

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Symantec applied a patch to our system that ensures that the JOI fields of sibling certificates exactly match the same fields in the final validated version of the original certificate.

Comment 1

[Jeremy Rowley]

No new info on this. How would you like us to proceed?

Comment 2

[Gervase Markham [:gerv]]

If the systems are patched (and are these systems now obsolete anyway?) and the certs are revoked, I think there's no further action.

Gerv