Closed Bug 1413761 Opened 2 years ago Closed 2 years ago
Digicert/Symantec: EV JOI Issue
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 Steps to reproduce: I wanted to report the following mis-issuance of EV certificates. I didn't find a report from Symantec from back when this happened, but I apologize if it's duplicating another bug. Note this only applies to the systems we acquired from Symantec, not DigiCert's (which means the system supporting the Sub CA plan is not at issue). I thought I'd post all of the issues we discover that weren't already closed to keep things transparent, even if older. The information is pretty old, but I'll have the team dig up any addition info required. 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. On July 20, 2017, a validation analyst identified and reported an existing EV certificate with incorrect Jurisdiction of Incorporation (JOI) fields. Other certificates were discovered through addition investigations. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. a. Certificate discovered with incorrect JOI on 2017-07-20 b. Investigation begins at 2017-07-21 to determine if the one occurrence reflected a systemic problem or a personnel training deficit. Initial review suggested issued was not reproducible. Case was left open. c. Follow up on the case was escalated 2017-10-04 19:39 PDT d. Cause was identified and verified 2017-10-05 04:45 PDT e. Root cause analysis started 2017-10-05 11:07 PDT f. Software patches were applied to our system on 2017-10-05 to correct the source of JOI information used, 2017-10-10 to address an edge case detected after 2017-10-05, and 2017-10-25 to introduce an additional safeguard by forcing certificates into manual review that otherwise would be issued via automation through reliance on data reuse rules. g. Initial extract of candidate certificates from our database on 2017-10-05 to locate certificates with similar mis-issuance. We further refined extracts through 2017-10-13. h. Analysis of all candidates completed 2017-10-16 16:45 PDT based on extracts occurring between 2017-10-05 and 2017-10-13. i. Revocation of impacted certificates occurred within 24 hours of problem confirmation in each specific certificate and occurred between 2017-10-10 and 2017-10-17 16:45 PDT 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. Symantec stopped issuing certificates with the problem as of 2017-10-10. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. Symantec found a total of 387 EV certificates with incorrect JOI fields. The first was issued 2013-03-11, and the last was issued 2017-10-10. The 108 still valid certificates were revoked. The remaining 279 certs had already been revoked or were expired. 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. They are EV certs so all of them should be in CT logs. The serial numbers are: 0b87cfc6300ac5e5e413bdd7fda91b49 79b46cf0e2b12bbe24cdf8191ea0fd1c 1a3b02cb7d69e6b9473ad85448e910ee 6ffab89edbf1c4fef8744524f96f8f7b 07f3f45a72f682989c4343fced576a7c 678852d5cc1ff1948cb1ab12cdf71249 51db3c9040f3911f6ec7d0c0e3dd8816 053e8cbd0e813907aef1d08a5d06c6b7 406874efb9ceb7d39a0de3a35e30e111 6de8a6771acd5554a5ca160101c13114 1f873ebfe850dfa153211cd928cfb565 3351836924dcd3026bb534b0bc17ef27 3666b4fc117f873d3b836722a7fea49f 7a163097a6bb2f750057583afa1b6d0a 2f8c502085c13127455ba4c5e34c0b91 3965928dae4f4c00e4d56815479a0722 20be9c85b6599c5ceb4db5ff5d7b946a 7a8ee91821eda10c429fd738f6e83202 055083056c525d14a9914e7400cd8bc3 7b9f0a59b1ca08dffdb2f89dfbaabfb8 4171f77eeebd6b0b53f8eba9361c1ef5 7a648d6d4bebe9cf9361c6833f78dfbd 46891614f5dc87092a4221b6df2b157a 395d1b5bf3f709b47056828a5463dd7d 03ceb22e0ed816d5284414c8b37c7179 5cd51da9d39570fcd2fe6a020a8c6171 7f9db327e5b236391f31da61477a59b8 46c8a7c8ce35116d9a06de4dca6e5380 5fe9d19c435fb89c0d753c6d26d19b93 0cd7dc8cfaadd902bee81125455dd636 16344550954324baa78d3392efdb83c7 5986b2f01248f7cd7875a535e6a2f9d5 0e8550b6ff1f12dcf1a560978275f371 533852556e1aae93a2ca063631aa499c 4c081134a842ad09f85ea90d0270751e 6f6aad895eb65c8fbe12c1990e6ced76 18b144319290b9d7b0e519fd0ccc4331 62c864954844965ca2f433431fb568c9 27617b1e86a51200f53529cc2731cb52 58336f7f8a25fb18222e75cb431389a1 1a39d61e38336eaad8435c2f605aa987 027f7dacf035aa80522ae7ba80fdb5b0 342ed8fcab5f472b481b8e565e5cf769 117a4e73f669bd3b0b145581087bb4f0 1b7d71985d2480d533761f87cb274b10 15a93838a97ad113b39d434f5cc5be29 485e8b5d2ebbb4ab963e0605ed949633 418c32860b574a13cd9c6862e38e134d 6fcc37f2403aa23e835b420a25df5409 7f70ef9e29489e185737458f6c38d2b2 472f317a9dbe53b557e5e6633e707b5b 7da921092b24d06c8fac19c4047e4c75 1d838eb488d57b2c9c8d9d1a7614c7e3 4d77a5e5a3f877aac74a333b7c82ebf9 0c0122b7dc928cdf11bde7c120a5aa80 302a98dc1d474405c9c082652e11bd61 1c1f74ea320bce810a16c96522675e45 1a7863eb14faf2d8d406fbbcb6239e3a 792040f434cb3dbfd4012e974120a944 01eeb9b156c060d50ca5cab7ddf5c8fe 4907fc19c6910359e150252a849319cc 6dd3e48c0be6994171e911da18e77f05 13ffa1e3c075584500efe42a4f9206be 3c83757d2db3d2f7c313d7c6f4b3dbf1 6f92781253cf0811807fa53189e0116a 3c8000f9715da94d01a9c2bebe24b156 5ea8df1f60c618313670cd6cd40a649d 69d83063680cf6142c41d538009c7aa2 555f1d3d6b98312cddc4018562c25450 582f53a13f2b5edfb4051dfdf7ae0c79 22d2dce5136d3fe1abb4c6b61e039d63 41778e580fec2bbd74dc067cc59ac962 3985165aa81ade7294b365e81b1e7721 73323e6344764ffecec9b7de517275ab 670d27a72a5fc599b90f98f1fb7854c0 0c57d7db2775a5b932b81a5704ea0676 3eef7434aa529e66992258a9fdd4f09e 1d90223e067adf6b0e18b94012b5c007 118ea8a4be4154c77d46018a3cb0efc6 4634980dfcb9bf2fd4e67ef2e260d979 054ee3b4b26a82ccadb4e802a524d38f 6119337e9b8e68a9ad3c6b0879f9f6f2 2b9350c978f63408da32dd7d9020c263 06073187074464edc6848e3abf0e154d 5fc246861883780e7ee78790573e7483 5fbcf64750a7ed75f76255398092ccb1 27e21bf630866be5cdb21dc1c9cdaa55 7bd81b30f5e22d69f23753a592458c7d 36f846397abe96071255772687b65fa3 152350a23448fb13742c42f35844ac57 0f52b7c9c0c15b0bc5a96c102d02d251 22e9ff427e944f4ec0266e0a61091dd9 1efa3036a59ab0088cea6367e32b023d 0d337ec2e534836de1a8b8c02d3ff1cf 349d716bef139d13b95391069fd4b26a 22d8ce5074a5488bf290a572421d71ef 4ae3f487c65b05915f52a0bba77c4dd9 29fbf0608108132732adf1d23010e975 705be4e68381875ba4df49ef13be77fa 1b73fd72e0c3f83d02ba7988d0ba04c5 09ea44ee9f94ca1651d58af16d2a4ed4 124b531ebe32a41ead6ce797ec8c293c 671fe934eaaeec44fde9f073695edd81 5002325b0a4501c3db858ece96bf6a11 76ef41cee360a25d5d9b6c4e2c773ae1 7eb40186398e3086cb91fd154d1d37dd 5fb9313ae1eb543585714110f6d066fe 5e05f8b9a4fcfc2ce3b111aeffe42a94 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. Symantec offered certificates containing, and signed with, RSA, ECDSA and DSA keys. Some customers received “sibling” certificates, in which the customer would purchase a certificate with one type of key and then request one or two siblings with keys from different PKI algorithms. The intent was that the sibling certificates would contain different keys yet identical Distinguished Name information, but the team discovered that in some cases, sibling EV certificates contained incorrect JOI fields. When Symantec received an order for an EV certificate, the cert was saved in the appropriate database, and pre-populated the JOI fields in the database from the Country, State and Locality of the Corporate Contact, aka Certificate Approver. During the normal course of validation in some cases as the team investigated the EV applicant, they determined that pre-populated JOI data was incorrect, they corrected this information, and saved it into the database. The dev team discovered a bug in its code in which the validated JOI fields were included in the original EV certificate, but the siblings contained JOI fields based on the initial Country, State and Locality of the Certificate Approver. The bug was introduced due to an oversight by a developer who failed to capture the case where initial placeholder JOI information could change during EV certificate validation requiring that the sibling be populated by the validated result rather than the initial placeholder data. The system also lacked an automated test case for JOI fields that differed from the Country, State and Locality of the Corporate Contact. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. Symantec applied a patch to our system that ensures that the JOI fields of sibling certificates exactly match the same fields in the final validated version of the original certificate.
Summary: EV JOI Issue → Digicert/Symantec: EV JOI Issue
No new info on this. How would you like us to proceed?
If the systems are patched (and are these systems now obsolete anyway?) and the certs are revoked, I think there's no further action. Gerv
Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.