Closed Bug 1389172 Opened 8 years ago Closed 8 years ago

DigiCert: Certificate Issues Identified on the Mailing List

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: kathleen.a.wilson)

References

Details

(Whiteboard: [ca-compliance] [uncategorized])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36 Actual results: The following issues were discovered through crt.sh by the Mozilla community and reported on the Mozilla dev-sec mailing list: 1) Serial Numbers less than 64-bit entropy a) InfoCert 179801356 47922359 72065818 b) Siemens 45010775 45000519 45073660 40700407 48827344 40726549 45068282 48827312 48726324 48726344 48726335 45068035 45060262 41273940 179803714 48827347 48827332 48827315 48827348 48726294 45029804 45053316 45061425 43151162 43151152 43151164 45060837 54576144 48881965 48726336 44438214 47906173 45048889 45257938 81268921 45257935 133008905 132946615 48726297 47884474 46110537 47211018 42791689 44377709 91004967 47612195 47646461 47612190 48827338 48715703 48827336 48726376 48773656 47743543 48773648 48773658 48773657 54576085 48773662 47605232 48827352 48773645 49706248 48827353 179801334 48773663 49706251 54576119 54576115 47939190 48827317 54397723 47815889 53205222 54576127 50756547 48773638 50165772 50756565 54576082 95852573 102169703 2) Metadata in OU field a) CTJ 53176869 57830766 65616693 117197299 93295619 104632894 21853640 22198796 30236343 30547236 32078607 34809638 39655364 40493955 43400295 45247984 47645844 45815760 40840774 48674276 40627018 48832585 57830776 57830777 57830779 57830781 61135216 57785675 57785694 53176900 53176901 53176903 64152048 78648920 53176895 53176898 57830773 102832536 51567050 132148480 110336219 111461024 80230338 81246678 91907174 91907175 110354478 110354479 78766827 79544403 78494050 80022464 15522999 16058171 20501110 20501178 21202825 21202827 21202829 20501158 20501137 20686696 20501187 20501201 20501217 20686693 20835061 21202831 21202833 21202835 21202836 26829282 11658435 11723002 11020877 11758671 12502488 12502489 13247204 13247205 13388567 42698962 16519562 19240755 20479086 20103452 21192792 54658744 21477087 42505252 29807378 57881547 30238735 30238738 30238741 30351648 47300904 54163587 54163620 54163639 51576885 56762416 70713886 73109181 78277867 73090743 77286114 75591249 b) DigiCert 28838043 29719356 41854275 45002523 42416834 42416853 42416863 44421681 45060999 43131232 43131307 43131498 43132441 48279184 48558476 48558484 48908976 58878082 58931131 68837112 71615331 71632071 78328813 49832975 74933357 76418106 76418197 76590177 76639330 76685188 76685195 81975356 82029229 97931808 97931812 106177929 138900619 32267988 41751647 105794465 15326026 47930798 45048314 10859239 19505358 98797497 98797525 17719410 47773488 40937363 48678004 105227697 83837393 27797536 42000357 42281109 42416587 48992066 49529547 64029766 42440045 23016305 42394669 72557347 165031011 c) Terena 32142920 20475084 11984821 23387766 40944750 50979270 66713765 73234645 99764268 134328239 3) Serial Number > 20 Octets a) TI Systems 176067928 160949550 160813046 166543375 173165333 168397193 168043243 179755654 176673561 179547720 165041960 179791026 173821648 161488914 173821651 165508757 179756125 179790644
Info Cert was revoked Aug 1, 2017. They shouldn't be a problem. We're reaching out to the remainder to have the certs revoked and replaced. For the CAs: 1) Siemens is no longer issuing certificates under the DigiCert hierarchy. Per our previous discussions, we are maintaining the CA solely to support previously issued certificates. Once all certs expire, the Sub CA will retire. 2) TI Systems is migrating away from a sub CA. We hope to complete the transition by the end of the year. 3) CTJ is migrating to a new platform. The transition will be lengthy, about a year. Jeremy
Additional certs 4) Local and test names a) Verizon https://crt.sh/?id=33626750&opt=cablint https://crt.sh/?id=12344381&opt=cablint 5) commonName not in SANS a) TI Trust Systems crt.sh/?id=100738527&opt=cablint crt.sh/?id=113547920&opt=cablint crt.sh/?id=113197913&opt=cablint crt.sh/?id=128965579&opt=cablint crt.sh/?id=139455685&opt=cablint b) Swiss Government https://crt.sh/?id=108003741&opt=cablint https://crt.sh/?id=108003737&opt=cablint
Blocks: BR-Compliance
Summary: Certificate Issues Identified on the Mailing List - DigiCert → DigiCert: Certificate Issues Identified on the Mailing List
Whiteboard: [ca-compliance]
6) Reserved IP Address: a) CTJ 33034069 b) TI Trust 10620806 crmaffari.telecomitalia.local crmaffari.telecomitalia.local , crmaffari 4 days ago 2017-11-18 10632702 com.telecomitalia.it com.telecomitalia.it 4 days ago 2017-12-17 10632711 allintranet.telecomitalia.it allintranet.telecomitalia.it 4 days ago 2018-06-23 13421134 oaforms.telecomitalia.it oaforms.telecomitalia.it , SVOAESEWEBc24 4 days ago 2019-01-14 12819786 oaforms.telecomitalia.it oaforms.telecomitalia.it , oaformsmanager.telecomitalia.it , oaformsapp.telecomitalia.it , helpme.telecomitalia.it , attivatore.telecomitalia.it , SVOAESEWEBc24 4 days ago 2019-02-03 7) Invalid dNSNnames a) TI Trust 14568767 10620806 10620776 10620792 9864794 9871186 13421134 12819786 b) Justica 10620923 c) WellsSecure 19560142 8) Bad Characters a) Wells Fargo DigiCert 19558707 2015-11-25 11:32:15 2016-11-24 11:32:15 ceomobilefix.wf.com DigiCert 11382596 2015-11-25 11:31:17 2016-11-24 11:31:17 ceomobile.wf.com 9)Double dot characters a) Inteso San Paulo https://crt.sh/?q=2B95B474A2646CA28DC244F1AE829C850EA41CF64C75E11A94FE8D228735977B&opt=cablint,x509lint
Here's the proposed resolution for each of these issues: 1a) InfoCert's subCA was revoked. 1b) We are working towards revocation of the Siemens Sub CA. We are still working out the exact date with them, but we currently plan to add them to OneCRL on Sep 1 and revoke on Sep 15. 2) Both CTJ and DigiCert have updated their systems to prevent metadata in the OU. Per the previous discussion on the issue, we don't plan to revoke the existing ones. (Source: https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/DgeLqKMzIds/E9--gj5WEAAJ). 3a) We plan to revoke the TI Trust issuing CA. We are still working out the exact date with them, but the current plan is to add them to OneCRL on Sep 1 and revoke on Sep 15. 4a) Both certificates are revoked. We've asked Verizon for a more detailed report on why these were missed previously. 5a) We plan to revoke the TI Trust issuing CA. We are still working out the exact date with them, but the current plan is to add them to OneCRL on Sep 1 and revoke on Sep 15. 5b) Both are revoked
6a) Revoked 6b) TI Trust systems is being revoked 7a) TI Trust systems is being revoked 7b) Justica revoked the cert 7c) Wells Fargo revoked the cert 8a) These were expired 9a) Revoked
To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information: 1) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below. - With the exception of the metadata issue, DigiCert is not issuing any of these certs. We've asked each of the Sub CAs to confirm they have updated their systems to prevent further mis-issuance. All confirmed they will fix the problem. Siemens and TI Trust provided unsatisfactory answers and are being revoked. In each communication, we stressed the need to revoke within 24 hours of receiving a problem report and provide a detailed remediation plan. 2) Explanation about how and why the mistakes were made, and not caught and fixed earlier. - Most of these certificates are fairly old. With the exception of reserved names, I think most CAs didn't think revoking the existing certs was a big deal as long as they fixed their system. 1) Serial Numbers less than 64-bit entropy a) InfoCert - revoked, issued due to older system b) Siemens - planned for revocation, no long using the intermediate and are operating in maintenance mode only. 2) Metadata in OU field - Patched during the previous discussion about metadata, but the decision then was revocation is not necessary. 3) Serial Number > 20 Octets - TI Trust is being revoked as they are unable to fix this system. 4) Local and test names - Verizon patched their system. We're waiting to hear why it was not caught and revoked earlier. 5) commonName not in SANS a) TI Trust Systems - Revoking b) Swiss Government - Patched earlier. 6) Reserved IP Address: a) CTJ - This cert was actually properly issued but wasn't revoked as part of the internal name revocation of 2016. They simply forgot to revoke this one. b) TI Trust - Being revoked 7) Invalid dNSNnames a) TI Trust - Revoking b) Justica - Asking for an explanation c) WellsSecure - Revoked. Asked for an explanation. 8) Bad Characters a) Wells Fargo - Patched and revoked 9)Double dot characters a) Inteso San Paulo - Revoked. I believed they patched their system, but I'll double check. 3) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. a) Revoking TI Trust and Siemens by Sep 15 b) Patching our systems and requiring the Sub CAs to patch their systems - already done c) Per our previous conversation, we are planning on requiring each CA to use cablint to find and detect errors prior to issuance. We're still working towards that goal by year's end. 4) Regular updates to confirm when those steps have been completed. a) Will do. Let me know if there are questions.
Telecom Italia Trust Systems has revoked some of the certificates listed in this bug. Siemens has indicated it is working on revoking some too. Both have indicated that the Sept. 1 (OneCRL) and Sept. 15 (revocation) are too aggressive considering the impact that it will have on their systems. Siemens is launching a new system on Sept. 8 that will CT log all certificates and issue certificates with serial numbers of appropriate length. We are working with Telecom Italia to create two new CAs (server and client) this week that will be housed inside DigiCert's infrastructure. They currently have approximately 2,200 server certificates and 13,000 client certificates that will need to be transitioned over.
(In reply to Jeremy from comment #6) > 3) List of steps your CA is taking to resolve the situation and ensure such > issuance will not be repeated in the future, accompanied with a timeline of > when your CA expects to accomplish these things. > a) Revoking TI Trust and Siemens by Sep 15 > b) Patching our systems and requiring the Sub CAs to patch their systems - > already done > c) Per our previous conversation, we are planning on requiring each CA to > use cablint to find and detect errors prior to issuance. We're still working > towards that goal by year's end. Can you provide a more descriptive timeline, enumerating which CAs (all sub-CAs or just those identified here), along with expected completion dates for such remediations? > 4) Regular updates to confirm when those steps have been completed. > a) Will do. Let me know if there are questions. Thanks for proactively identifying and responding to the questions that have been directed at other CAs regarding these misissuances. I'll note that this reply was over a week ago, and a number of items were listed as "in progress" with limited to no timeline detail. Can you please review the answers you've provided, so that we can ensure we have a complete and timely understanding? (In reply to Ben Wilson from comment #7) > Telecom Italia Trust Systems has revoked some of the certificates listed in > this bug. Does this mean that it has not revoked others? Please provide further details, such as what certificates have _not_ been revoked. > Siemens has indicated it is working on revoking some too. Similarly, can you please indicate which certificates have _not_ been revoked. > Both have indicated that the Sept. 1 (OneCRL) and Sept. 15 (revocation) are too > aggressive considering the impact that it will have on their systems. - Do you believe this is consistent with DigiCert's CP/CPS? - Does your CP/CPS describe the process that such 'leeway' is granted if something is reported by the customer as 'too aggressive' - Do you have reason to believe these two CAs comply with all aspects of the Baseline Requirements and Mozilla inclusion policy, such that failing to revoke does not introduce further or additional risk? Please provide details supporting this, if so. > Siemens is launching a new system on Sept. 8 that will CT log all > certificates and issue certificates with serial numbers of appropriate > length. Does this mean that, until then, Siemens will continue to violate the BRs and does not have remediation? As previously indicated, these CAs provided "unsatisfactory answers" (which are undisclosed on this bug), and so this does not seem a reasonable mitigation. > We are working with Telecom Italia to create two new CAs (server > and client) this week that will be housed inside DigiCert's infrastructure. > They currently have approximately 2,200 server certificates and 13,000 > client certificates that will need to be transitioned over. It does not appear, for purposes of OneCRL, that there is a need to transition the 13,000 client certificates in order to ensure a smooth transition. Can you provide a schedule for when Telecom Italia will have servers transitioned (both when it expects to first be able to transition and when it expects to last complete that transition), so that Mozilla can make an informed decision about the compatibility impact of adding to OneCRL.
Hey Ryan, Here's a more definitive timeline: 1) Siemens - Revoking the Sub CA. The date is still set to Sep 15, but I'll let Ben manage the conversation. The mis-issued certificates are still not revoked. 2) InfoCert - already revoked 3) TI Trust - TI Trust systems successfully revoked all of their certificates. They are migrating away from the Sub CA and asked for more time. Ben is working with them to establish a date. 4) Verizon - patched. These were actually mis-issued pre-certs. 5) Well's Fargo - revoked and patched 6) CTJ - patched already 7) Swiss Gov - patched already 8) Justica - patched alreaduu 9) Inteso San Paulo - revoked the cert Using cablint for every certificate will be required by the end of the year. We've asked all Sub CAs to provide a complete repository of certificates. These will be scanned through cablint for issues. Some have started delivering. We expect a complete set by Oct. (Sorry for the delay in updates - I was gone most of last week)
(In reply to Jeremy from comment #0) Here is an updated status on revocation: > > 3) Serial Number > 20 Octets > a) TI Systems > 176067928 Revoked > 160949550 Revoked > 160813046 Revoked > 166543375 Revoked > 173165333 Revoked > 168397193 Revoked > 168043243 Revoked > 179755654 Revoked > 176673561 Revoked > 179547720 Revoked > 165041960 Revoked > 179791026 Revoked > 173821648 Revoked > 161488914 Revoked > 173821651 Revoked > 165508757 Revoked > 179756125 Revoked > 179790644 Revoked
(In reply to Jeremy from comment #2) Here is a revocation status of the following certificates: > Additional certs > 5) commonName not in SANS > a) TI Trust Systems > crt.sh/?id=100738527&opt=cablint Revoked > crt.sh/?id=113547920&opt=cablint Revoked > crt.sh/?id=113197913&opt=cablint Revoked > crt.sh/?id=128965579&opt=cablint Revoked > crt.sh/?id=139455685&opt=cablint Revoked > > b) Swiss Government > https://crt.sh/?id=108003741&opt=cablint Revoked > https://crt.sh/?id=108003737&opt=cablint Revoked
(In reply to Jeremy from comment #3) Here is the revocation status of the following certificates: > > b) TI Trust > 10620806 crmaffari.telecomitalia.local crmaffari.telecomitalia.local , > crmaffari 4 days ago 2017-11-18 Revoked > 10632702 com.telecomitalia.it com.telecomitalia.it 4 days ago 2017-12-17 Revoked > 10632711 allintranet.telecomitalia.it allintranet.telecomitalia.it 4 days > ago 2018-06-23 Revoked > 13421134 oaforms.telecomitalia.it oaforms.telecomitalia.it , SVOAESEWEBc24 > 4 days ago 2019-01-14 Revoked > 12819786 oaforms.telecomitalia.it oaforms.telecomitalia.it , > oaformsmanager.telecomitalia.it , oaformsapp.telecomitalia.it , > helpme.telecomitalia.it , attivatore.telecomitalia.it , SVOAESEWEBc24 4 > days ago 2019-02-03 Revoked > > 7) Invalid dNSNnames > a) TI Trust > 14568767 Revoked > 10620806 Revoked > 10620776 Revoked > 10620792 Revoked > 9864794 Revoked > 9871186 Revoked > 13421134 Revoked > 12819786 Revoked
(In reply to Ryan Sleevi from comment #8) > (In reply to Ben Wilson from comment #7) > > Telecom Italia Trust Systems has revoked some of the certificates listed in > > this bug. > > Does this mean that it has not revoked others? Please provide further > details, such as what certificates have _not_ been revoked. I am continuing my review of Telecom Italia Trust Technologies, but I may have been mistaken in saying "some". So far, all of the certificates issued by them in this bug were revoked a couple of weeks ago. > > > Siemens has indicated it is working on revoking some too. > > Similarly, can you please indicate which certificates have _not_ been > revoked. I am continuing my review and will have more information next week. So far, the list of certificates that I am relying on is in https://bugzilla.mozilla.org/attachment.cgi?id=8898848 under the tab "SSL CA 2013 ZZZZZZY9". I have a call with Siemens on Monday and will have more information then. > > > Both have indicated that the Sept. 1 (OneCRL) and Sept. 15 (revocation) are too > > aggressive considering the impact that it will have on their systems. > > - Do you believe this is consistent with DigiCert's CP/CPS? I believe that working with Siemens and Telecom Italia on this issue in the manner currently undertaken is consistent with the DigiCert CP/CPS and good CA practice. > - Does your CP/CPS describe the process that such 'leeway' is granted if > something is reported by the customer as 'too aggressive' As stated above, this is consistent with our CP/CPS and good CA practice. > - Do you have reason to believe these two CAs comply with all aspects of the > Baseline Requirements and Mozilla inclusion policy, such that failing to > revoke does not introduce further or additional risk? Please provide details > supporting this, if so. I believe that in some cases a CA could be justified in not revoking certificates within a 24-hour timeframe. Risks and circumstances should be considered. I do not have knowledge of "all" aspects as they relate to the Baseline Requirements or the Mozilla inclusion policy. However, I have no additional basis, other than these misissuances, to believe that the CA or the failure to revoke all of these certificates within 24 hours poses additional risk. > > > Siemens is launching a new system on Sept. 8 that will CT log all > > certificates and issue certificates with serial numbers of appropriate > > length. > > Does this mean that, until then, Siemens will continue to violate the BRs > and does not have remediation? As previously indicated, these CAs provided > "unsatisfactory answers" (which are undisclosed on this bug), and so this > does not seem a reasonable mitigation. Previously and separately elsewhere, Siemens has explained that it has taken its CA offline while it is working to remediate the problem with its serial numbers. I have a call with Siemens on Monday and will have more information then. As also noted elsewhere, Siemens stopped issuing under the Siemens Issuing CA Class Internet Server 2013 last year. > > > We are working with Telecom Italia to create two new CAs (server > > and client) this week that will be housed inside DigiCert's infrastructure. > > They currently have approximately 2,200 server certificates and 13,000 > > client certificates that will need to be transitioned over. > > It does not appear, for purposes of OneCRL, that there is a need to > transition the 13,000 client certificates in order to ensure a smooth > transition. Can you provide a schedule for when Telecom Italia will have > servers transitioned (both when it expects to first be able to transition > and when it expects to last complete that transition), so that Mozilla can > make an informed decision about the compatibility impact of adding to OneCRL. Yesterday we created two CAs for Telecom Italia-- https://ccadb.force.com/0011J000019xflQ and https://ccadb.force.com/0011J000019xeiw. These two CAs are now being tested by Telecom Italia. Until I speak with Telecom Italia next week, I do not have any further information on their SSL/TLS Server Certificate replacement efforts.
This morning we had a call with Siemens. They are re-checking everything on their 2017 CA and are in communication with their auditor before bringing the 2017 CA live on 8-Sept-2017. Meanwhile they are replacing certificates with ones issued by Symantec. They will continue this work throughout September, when they expect to have all certificates replaced/revoked. This effort requires that they contact their internal customers. Some customers have restrictions on allowed downtime. Others have implemented certificate pinning, making certificate replacement more difficult. Siemens' process involves identifying and mitigating potential risks that would otherwise be encountered by moving too fast. They are prioritizing the revocation and replacement of the 137 DigiCert-chained certificates with a goal to have those done by September 15. Measures being implemented to prevent/avoid this situation in the future include: increasing the formal level of review of all changes proposed by browsers/CABF; immediate re-audits based on the adoption of new requirements; and conducting internal lessons-learned workshops. They will be providing DigiCert with a weekly update to their revocation tracking spreadsheet until all certificates are revoked and replaced in September.
The plan is still to add the Siemens intermediate to oneCRL by September 15th.
Attempting to summarize the status below: Because there's such a large number of issues here, particularly with sub-CAs of DigiCert, this is a bit difficult, particularly as new issues emerged. I've attempted to break this down per-subCA, which may be appropriate to do so in the future. Note that when I say "Remediation not performed", I am explicitly excluding the notion of simply revoking the certificates. Rather, we want to see a comprehensive root cause analysis as to what caused the issue, and what systems adn controls are in place. Unless DigiCert is planning to revoke these certificates within 7 days, I would further suggest that DigiCert must, for each of these cases, perform their own post-mortem regarding their supervision and oversight of their subordinate CAs, and what steps they have or are taking to resolve these issues. Kathleen, I suspect that given the number of issues, and the number of sub-CAs for which we are lacking meaningful response or analysis for, that we probably should be looking at creating a bug for each and every one of these issues (e.g. letter and number) to allow DigiCert the opportunity to respond with the 7 questions being asked. 1) InfoCert a) Insufficient Serial Number Entropy - See Comment #0, Comment #4, Comment #6 - Remediation - 2017-08-01 - CA Certificate was Revoked 2) Siemens a) Insufficient Serial Number Entropy - See Comment #0, Comment #1, Comment #4, Comment #6, Comment #9, Comment #13, Comment #14, Comment #15 - Remediation - 2017-09-15 - CA certificate will be revoked 3) CTJ a) Metadata in OU fields - See Comment #0, Comment #1, Comment #4, Comment #9 - Remediation - 2017-08-16 - CTJ systems patched - 2018-08-XX - CTJ migrates to a new system (See Comment #1) b) Reserved IP Address - See Comment #3, Comment #6, Comment #9 - Remediation - None performed; no root cause analysis performed 4) DigiCert a) Metadata in OU fields - See Comment #0, Comment #4, Comment #6 - Remediation - 2017-08-16 - DigiCert systems patched 5) Terena a) Metadata in OU fields - See Comment #0 - Remediation - None performed; no root cause analysis performed 6) Telecom Italia Trust Systems (TI Trust Systems / TI Systems) a) Serial numbers > 20 octets - See Comment #0, Comment #1, Comment #4, Comment #5, Comment #7, Comment #9, Comment #10, Comment #13 - Remediation - None performed. - Comment #4 and Comment #5 proposed revoking this sub-CA, but Comment #9 walked this back and indicated DigiCert would not be revoking. - Comment #7 suggests standing up a new sub-CA, and expanded upon in Comment #13. Neither of these remedy the problems with the existing CAs. b) Common Names not in SANs - See Comment #2, Comment #3, Comment #6, Comment #9, Comment #10, Comment #13 - Remediation - None performed. See above. c) Reserved/Intranet domain name - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13 - Remediation - None performed. See above. d) Invalid DNS names - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13 - Remediation - None performed. See above. 7) Justica a) Invalid DNS names - See Comment #3, Comment #5, Comment #6, Comment #9 - Remediation - 2017-08-24 - CA patched (See Comment #9) 8) Wells Fargo a) Reserved/Intranet domain name - See Comment #3, Comment #5, Comment #6, Comment #9 - Remediation - 2017-08-24 - CA patched (See Comment #9) b) Invalid DNS name (trailing whitespace) - See Comment #3, Comment #5, Comment #6, Comment #9 - Remediation - 2017-08-24 - CA patched (See Comment #9) 9) Swiss Government a) CommonName not in SANs - See Comment #2, Comment #4, Comment #6, Comment #9 - Remediation - 2017-08-24 - CA patched 10) Verizon a) Reserved/Intranet domain name - See Comment #2, Comment #4, Comment #6, Comment #9 - Remediation - 2017-08-24 - CA patched 11) Inteso San Paulo a) Double dot characters - See Comment #3, Comment #5, Comment #6, Comment #9 - Remediation - None performed Overall, I think the level of communication and detail here is far below the standard of what we'd expect and are expecting of CAs, but the depth and breath of misissuance by DigiCert's subordinate CAs, and the sheer volume of information here, have prevented effectively enforcing this. For this reason, I suggested the above split-out, to allow each of these issues to be followed up on. On the positive side, a number of these issues were self-reported (See Comment #0, Comment #2, Comment #3) or attempting to be proactive (See Comment #6). This is positive and encouraging, although it's expected of all CAs as a minimum standard. On the negative side, multiple plans for remediation have failed to be met, most notably with both Siemens and TI Trust Systems, which highlights both the importance of managing these as explicit issues and concerns with timelines DigiCert proposes to respond. Kathleen, Gerv, I'm flagging this for you to review and to determine how to proceed effectively.
Flags: needinfo?(kwilson)
Flags: needinfo?(gerv)
This morning we had a call with Telecom Italia. They are no longer issuing certificates from the TI Trust Technologies Global CA (the “TI Trust CA”). Instead, their SSL/TLS Server Certificates are being issued from DigiCert’s infrastructure under the Trust Technologies Global CA, which is owned and controlled by DigiCert as a subordinate CA to the Baltimore Cybertrust Root, which is subject to ongoing WebTrust audit and compliance with the Baseline Requirements. As to the reason for the misissuances, they were apparently due to use of the Unicert CA platform, which is no longer being supported. They were not aware of the problem until notified by us. The misissuances were not detected as part of an annual audit because until recently these certificate criteria were not examined by auditors. Telecom Italia has already revoked the certificates with problems identified in my following responses: https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c10, https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c11, and https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c12. Telecom Italia now intends to revoke all previously issued certificates with the serial number problem (about 800-900) by 31 December 2017. Again, these were due to a bug or unsupported feature of the Unicert CA software. Telecom Italia have represented to us that they will accelerate certificate revocation/replacement as they become more familiar with our systems and API. Stressing the importance that their customers, including the Italian Government, not be unduly disrupted, they would like to avoid having the TI Trust CA being placed on OneCRL or revoked. They would like the 900 or so remaining SSL/TLS server certificates to be replaced in the normal course of business or when they expire. This will allow them to replace these otherwise-good certificates over time, as well as the 13,000 client certificates I mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1389172#c7.
Kathleen is creating a bug for each Sub CA. I'll be the owner on each bug and add the Sub CA's to the cc list. This way we can address each one separately.
(In reply to Ryan Sleevi from comment #16) > > 1) InfoCert > a) Insufficient Serial Number Entropy > - See Comment #0, Comment #4, Comment #6 > - Remediation > - 2017-08-01 - CA Certificate was Revoked Bug #1397951 > > 2) Siemens > a) Insufficient Serial Number Entropy > - See Comment #0, Comment #1, Comment #4, Comment #6, Comment #9, > Comment #13, Comment #14, Comment #15 > - Remediation > - 2017-09-15 - CA certificate will be revoked Bug #1397954 > 3) CTJ > a) Metadata in OU fields > - See Comment #0, Comment #1, Comment #4, Comment #9 > - Remediation > - 2017-08-16 - CTJ systems patched > - 2018-08-XX - CTJ migrates to a new system (See Comment #1) > b) Reserved IP Address > - See Comment #3, Comment #6, Comment #9 > - Remediation > - None performed; no root cause analysis performed > Bug #1397957 > 4) DigiCert > a) Metadata in OU fields > - See Comment #0, Comment #4, Comment #6 > - Remediation > - 2017-08-16 - DigiCert systems patched > Will keep this one in this bug, Bug #1389172 For the DigiCert "Metadata in OU fields" problem listed above, please provide an incident report as described here: https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report > 5) Terena > a) Metadata in OU fields > - See Comment #0 > - Remediation > - None performed; no root cause analysis performed Bug #1397958 > > 6) Telecom Italia Trust Systems (TI Trust Systems / TI Systems) > a) Serial numbers > 20 octets > - See Comment #0, Comment #1, Comment #4, Comment #5, Comment #7, > Comment #9, Comment #10, Comment #13 > - Remediation > - None performed. > - Comment #4 and Comment #5 proposed revoking this sub-CA, but Comment > #9 walked this back and indicated DigiCert would not be revoking. > - Comment #7 suggests standing up a new sub-CA, and expanded upon in > Comment #13. Neither of these remedy the problems with the existing CAs. > b) Common Names not in SANs > - See Comment #2, Comment #3, Comment #6, Comment #9, Comment #10, > Comment #13 > - Remediation > - None performed. See above. > c) Reserved/Intranet domain name > - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13 > - Remediation > - None performed. See above. > d) Invalid DNS names > - See Comment #3, Comment #6, Comment #9, Comment #10, Comment #13 > - Remediation > - None performed. See above. Bug #1397960 > > 7) Justica > a) Invalid DNS names > - See Comment #3, Comment #5, Comment #6, Comment #9 > - Remediation > - 2017-08-24 - CA patched (See Comment #9) Bug #1397961 > > 8) Wells Fargo > a) Reserved/Intranet domain name > - See Comment #3, Comment #5, Comment #6, Comment #9 > - Remediation > - 2017-08-24 - CA patched (See Comment #9) > b) Invalid DNS name (trailing whitespace) > - See Comment #3, Comment #5, Comment #6, Comment #9 > - Remediation > - 2017-08-24 - CA patched (See Comment #9) Bug #1397963 > > 9) Swiss Government > a) CommonName not in SANs > - See Comment #2, Comment #4, Comment #6, Comment #9 > - Remediation > - 2017-08-24 - CA patched Bug #1397965 > > 10) Verizon > a) Reserved/Intranet domain name > - See Comment #2, Comment #4, Comment #6, Comment #9 > - Remediation > - 2017-08-24 - CA patched Bug #1397968 > > 11) Inteso San Paulo > a) Double dot characters > - See Comment #3, Comment #5, Comment #6, Comment #9 > - Remediation > - None performed Bug #1397969
Flags: needinfo?(kwilson)
Flags: needinfo?(gerv)
(In reply to Kathleen Wilson from comment #19) > > 4) DigiCert > > a) Metadata in OU fields > > - See Comment #0, Comment #4, Comment #6 > > - Remediation > > - 2017-08-16 - DigiCert systems patched > > > > > Will keep this one in this bug, Bug #1389172 > For the DigiCert "Metadata in OU fields" problem listed above, please > provide an incident report as described here: > https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report This was originally noted as "Issue 2b" Jeremy, Ben - can one of you make sure there's a consistent reply for the incident report (mentioned with the above template)? I think many of the seeds of the information are here, I think if we can get a consistent report here we can look at closing it out. Thanks.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(jeremy.rowley)
1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date. - The issue was posted to the Mozilla dev list on Aug 9, 2017 2. A timeline of the actions your CA took in response. - We investigated to see what was going on that same day, responding to Jonathan. Because this is not a SubCA, we were able to post initial findings to that mailing list. We determined this was not a direct violation of the BRs as it states "All other subject attributes MUST contain information that has been verified by the CA". OU is not an "other subject attribute" and is defined in the section immediately prior to this. As such, we have not revoked these certificates. 3. Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem. - Confirmed. DigiCert has patched its systems to detect whether the OU field has metadata and prevents issuance if non-validated information is present. Although I don't think this is actually a violation per the above explanation, we do see value in preventing meta-data from all subject fields, not just other subject fields. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. - Unknown. Customers can enter data they want in the OU Field. These are screened for addresses and names. We continuously patch our systems as new forms of metadata are found. 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. 28838043 29719356 41854275 45002523 42416834 42416853 42416863 44421681 45060999 43131232 43131307 43131498 43132441 48279184 48558476 48558484 48908976 58878082 58931131 68837112 71615331 71632071 78328813 49832975 74933357 76418106 76418197 76590177 76639330 76685188 76685195 81975356 82029229 97931808 97931812 106177929 138900619 32267988 41751647 105794465 15326026 47930798 45048314 10859239 19505358 98797497 98797525 17719410 47773488 40937363 48678004 105227697 83837393 27797536 42000357 42281109 42416587 48992066 49529547 64029766 42440045 23016305 42394669 72557347 165031011 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. - OU is traditionally just a dumping ground for non-validated information. Although the BRs adopted the requirement that no other field should contain meta-data, the requirement does not squarely apply to the OU field under the language (as it's not an "Other Subject" field. However, rather than fight over nuances with respect to the OU, DigiCert patched the issue and put safeguards in place to prevent re-occurrence. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. - We add new metadata to a blacklist as it is discovered. We are also moving to a platform where all OU information is verified prior to issuance.
Flags: needinfo?(jeremy.rowley)
(In reply to Jeremy from comment #21) > 2. A timeline of the actions your CA took in response. > - We investigated to see what was going on that same day, responding to > Jonathan. Because this is not a SubCA, we were able to post initial findings > to that mailing list. We determined this was not a direct violation of the > BRs as it states "All other subject attributes MUST contain information that > has been verified by the CA". OU is not an "other subject attribute" and is > defined in the section immediately prior to this. As such, we have not > revoked these certificates. This isn't a timeline :) For example, this doesn't include the answer below: (In reply to Jeremy from comment #21) > 6. Explanation about how and why the mistakes were made or bugs introduced, > and how they avoided detection until now. > - OU is traditionally just a dumping ground for non-validated information. > Although the BRs adopted the requirement that no other field should contain > meta-data, the requirement does not squarely apply to the OU field under the > language (as it's not an "Other Subject" field. However, rather than fight > over nuances with respect to the OU, DigiCert patched the issue and put > safeguards in place to prevent re-occurrence. As captured in https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Examples_of_Good_Practice , can you provide a timeline (for example, both https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and https://groups.google.com/d/msg/mozilla.dev.security.policy/nMxaxhYb_iY/AmjCI3_ZBwAJ provide concrete dates and times for the set of changes made to the sytem)
Flags: needinfo?(jeremy.rowley)
Got it - Here's the timeline: 1. Aug 9, 2017 - Report was made to the mailing list 2. Aug 9, 2017 - DigiCert responded to mailing list and began investigation 3. Aug 10, 2017 - DigiCert decided this is not a violation as posted on the mailing list: Section 7.1.4.2.2(J) : "All other optional attributes, when present within the subject field, MUST contain information that has been verified by the CA. Optional attributes MUST NOT contain metadata such as ‘.’, ‘‐‘, and ‘ ‘ (i.e. space) characters, and/or any other indication that the value is absent, incomplete, or not applicable." Because OU is not "all other optional attributes", j doesn't apply. However, DigiCert decided that preventing metadata in the OU field was a good thing to do so committed to preventing it going forward. 4. Aug 10, 2017 - OU metadata was added to a blacklist on the CA side that prevented issuance 5. ETA beginning of Nov, 2017 - DigiCert will validate all OU information prior to inclusion. This will eliminate all future OU metadata that is not currently blacklisted.
Flags: needinfo?(jeremy.rowley)
Thanks. I believe we can close this bug out.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Want me to go add a similar timeline to the TERENA bug and others ones that are ready to close out? TERENA was patched at the same time (it's the same system as ours).
If the other bugs do not have complete details, please do update them :)
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [uncategorized]
You need to log in before you can comment on or make changes to this bug.