1390990 - D-TRUST: Non-BR-Compliant Certificate Issuance

Reporter

Description

•

8 years ago

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience. To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information: 1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date. 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below. 3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem. 4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued. 5) Explanation about how and why the mistakes were made, and not caught and fixed earlier. 6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. 7) Regular updates to confirm when those steps have been completed. Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states: “The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: … 9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; … 14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time). However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement. We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates. The problems reported for your CA in the mozilla.dev.security.policy forum are as follows: ** Failure to respond within 24 hours after Problem Report submitted https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ The problems were reported via your CA’s Problem Reporting Mechanism as listed here: https://ccadb-public.secure.force.com/mozilla/CAInformationReport Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this. ** dNSName containing '/' https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ ** Short / sequential-looking serial numbers https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ RESOLUTION: https://groups.google.com/d/msg/mozilla.dev.security.policy/UnR98QjWQQs/O-Hf5T4WBwAJ

Arno Fiedler

Assignee

Comment 1

•

8 years ago

Dear Forum, as already said, we really regret this delay in implementing the required change and this none- conformance to the CA/B-Forum and Mozilla requirements. The following shall amplify the background, facts and measures taken. Since we have two issues, we split our answers into A) and B). 1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date. - A) CN/SAN – mozilla.dev.security.policy 07-08-2017 - B) SerialNo – internal audit begin of July ----------------------- 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below. - A) CN/SAN – immediately at 07-08-2017 - B) SerialNo – at 07-07-2017 ----------------------- 3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem. - Detailed lists has been send via eMail to Mozilla: OV, EV, SAN (CSV file including SerialNo, date and sha1 fingerprints) - Today we have implemented CT for EV certificates. Since these are included into the production process before issuing the certs, the question is how to get already issued certs into the logs? - The OV CT implementation has been scheduled for September 2017. ------------------------ 4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued. - A) CN/SAN – number of certs: 1 - B) SerialNo – number of certs per CA o Start Date: 30-09-2016 o Issuing End Date EV: 15-05-2017: 63 EV TLS certs o Issuing End Date OV: 07-07-2017: 607 OV TLS certs ----------------------- 5) Explanation about how and why the mistakes were made, and not caught and fixed earlier. - A) CN/SAN o Individual failure though 4-eyes-principle: - This was the first time the mistake occurred - Not user-friendly production backend GUI o New Bug within our CSR Validator for Retail customers after system update - B) SerialNo o B1) We thought to be compliant: Based on technical, internal miss-understanding how to reach this 64 bit entropy in the certificate. We believed to be accepted compliant by using the DNqualifier in addition to the serialNo as a temporary solution. According to our platform-wise migration plan we completed the implementation finally at 15-05-2017. This kind of splitted migration was necessary, because of our different registration and rollout platforms for different customer groups (small retail to big enterprise solutions). During an internal audit at begin of July we realized, that we missed one customer platform for OV certificates. We stopped issuing these certificates on 07-07-2017 o B2) Because of dependency on 3rd party software and hardware vendors as well as customer contracts we had a delay with implementation in time. -------------------------- 6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. - A) CN/SAN o Immediate additional training, specific training on correct CN/SAN encoding o Immediate update on standard training materials o Update on CSR-Validator used during registration until 24-08-2017 (scheduled HotFix release) and optimization of production backend GUI o Revocation: 17-08-2017 12:00 PM - B) SerialNo o This has been already solved by 07-07-2017 o We will have a real-time dashboard showing issued certificates to fulfill nearly real time audits by the end of year. That should make it easier to look at the certs more effectively on relevant certificate content. o We will communicate possible delays of necessary changes in a timely manner and pro-actively to the rootstore program management. We already optimized and changed our e-mail POCs to make communication more reliable. -------------------------- 7) Regular updates to confirm when those steps have been completed. We will do so and publish a final status report until 15-09-2017. Our forthcoming annual audit in early October will list results of implementation of the finding in our CA’s BR audit statement as required. ------------------------ We hope that this report has answered your open questions effectual. If we forgot any open issue please do not hesitate to come up to us.

Ryan Sleevi

Comment 2

•

8 years ago

(In reply to Arno Fiedler from comment #1) > 5) Explanation about how and why the mistakes were made, and not caught and > fixed earlier. > - A) CN/SAN > o Individual failure though 4-eyes-principle: > - This was the first time the mistake occurred > - Not user-friendly production backend GUI > o New Bug within our CSR Validator for Retail customers after system update Can you expand on the answers here, as they may have been lost in translation. That is, what do you mean by production backend GUI? What was the BUG with the CSR Validator? A useful example would be to provide a timeline, from when the bug was first introduced (e.g. was it a specific version of software, was it always present, etc), when it was detected, when it was fixed, how it was fixed, what other tests have been introduced, etc. Understanding that human mistakes are inevitable, our goal is to look holistically at the process, and understand where failures were introduced (without assigning blame), and understanding what systemic steps can be taken. For example, the four-eyes-principle is an attempt to mitigate the risk of a single human judgement call/failure, and that's an important mitigation, but it's also useful to understand what technical controls could have existed, why they are (or are not) reasonable. In addition to understanding what steps are being taken to remediate the issue, our goal is to understand holistically what steps are being taken to detect, prevent, or mitigate such issues in the future. > > - B) SerialNo > o B1) We thought to be compliant: > Based on technical, internal miss-understanding how to reach this 64 > bit entropy in the certificate. > We believed to be accepted compliant by using the DNqualifier in > addition to the serialNo as a temporary solution. > According to our platform-wise migration plan we completed the > implementation finally at 15-05-2017. > This kind of splitted migration was necessary, because of our > different registration and rollout platforms for > different customer groups (small retail to big enterprise solutions). > During an internal audit at begin of July we realized, that we > missed one customer platform for OV certificates. > We stopped issuing these certificates on 07-07-2017 > o B2) Because of dependency on 3rd party software and hardware vendors as > well as customer contracts we had a delay > with implementation in time. Similarly, with B2, it's useful if you can share more detail, so that the Mozilla community can understand the challenges, and the overall ecosystem can better understand and respond to such issues in the future. For example, the answer in B2 suggests that if similar technical issues may arise, it may also take several months to remediate. As that would be quite concerning, it's useful to understand what changes have been made to ensure that 3rd party software, hardware vendors, or customer contracts can be changed more quickly in the future. Similarly, to mitigate the risk of post-deadline changes, it's useful to know what steps have been taken to ensure/review compliance and/or interpretations. For example, did you consult with your auditor regarding the DNqualifier, and did they advise you it would be compliant? If so, that's useful information to share. If not, that seems a useful addition to your process as changes come through the CA/Browser Forum. These are the types of holistic mitigations and understanding we're attempting here. > -------------------------- > > 6) List of steps your CA is taking to resolve the situation and ensure such > issuance will not be repeated in the future, > accompanied with a timeline of when your CA expects to accomplish > these things. > - A) CN/SAN > o Immediate additional training, specific training on correct CN/SAN encoding > o Immediate update on standard training materials > o Update on CSR-Validator used during registration until 24-08-2017 > (scheduled HotFix release) and optimization > of production backend GUI > o Revocation: 17-08-2017 12:00 PM Can you confirm your update is scheduled for this Wednesday? Just wanting to follow-up and make sure timelines are met :) > > - B) SerialNo > o This has been already solved by 07-07-2017 > o We will have a real-time dashboard showing issued certificates to fulfill > nearly real time audits by the end of year. > That should make it easier to look at the certs more effectively on > relevant certificate content. Can you expand this timeline more, beyond "by the end of the year"? Systemically, I think there are still some concerns with ensuring that we've analyzed the root causes, and ensured there's appropriate mitigations. I believe there's still opportunities for improvements here in the response, based on the information shared, and still some uncertainty needing to be addressed.

Kim Nguyen

Comment 3

•

8 years ago

Hello Ryan, many thanks for the valuable Feedback and comments, we are collecting and preparing further input and will reply as soon as possible, latest by end of this week, so that we can report an actual Status about the customer communication and the implementation of the Bugfix thus addressing the uncertainties you mention.

Ryan Sleevi

Updated

•

8 years ago

Flags: needinfo?(kim.nguyen)

Arno Fiedler

Assignee

Comment 4

•

8 years ago

Hello Ryan, we hope that we answer your open questions effectual. ----------------------------------- Can you expand on the answers here, as they may have been lost in translation. That is, what do you mean by production backend GUI? What was the BUG with the CSR Validator? A useful example would be to provide a timeline, from when the bug was first introduced (e.g. was it a specific version of software, was it always present, etc.), when it was detected, when it was fixed, how it was fixed, what other tests have been introduced, etc. ANSWER: The GUI of our validation software backend which our team is using, had some usability and visualization related issues. This implied, that the way multiple SANs were displayed had potential for mistakes. We released the improvement of the backend GUI yesterday as previously announced. The bug mentioned with respect to the CSR Validator was, that the CSR validator didn’t filter prohibited characters correctly and was introduced by the previous release but was not recognized during test. ----------------------------------- Understanding that human mistakes are inevitable, our goal is to look holistically at the process, and understand where failures were introduced (without assigning blame), and understanding what systemic steps can be taken. For example, the four-eyes-principle is an attempt to mitigate the risk of a single human judgement call/failure, and that's an important mitigation, but it's also useful to understand what technical controls could have existed, why they are (or are not) reasonable. In addition to understanding what steps are being taken to remediate the issue, our goal is to understand holistically what steps are being taken to detect, prevent, or mitigate such issues in the future. ANSWER: Retrospectively speaking, more efforts to testing and training could have avoided the software bug as well as the human failure. Hence, the combination of all above mentioned mistakes was the reason behind this. But after having analyzed the issues in detail we believe, that the quality and quantity of all controls is quite appropriate: e.g. CSR validator, 4-eyes-principal validation process (which is already implemented, regular and case based trainings, software testing and release management. Nevertheless we decided to extend our software testing taking the experiences of this incidents into account. ---------------------------- Similarly, with B2, it's useful if you can share more detail, so that the Mozilla community can understand the challenges, and the overall ecosystem can better understand and respond to such issues in the future. For example, the answer in B2 suggests that if similar technical issues may arise, it may also take several months to remediate. As that would be quite concerning, it's useful to understand what changes have been made to ensure that 3rd party software, hardware vendors, or customer contracts can be changed more quickly in the future. Similarly, to mitigate the risk of post-deadline changes, it's useful to know what steps have been taken to ensure/review compliance and/or interpretations. ANSWER: - 3rd party software / hardware: We are in discussions with our 3rd party vendors in order to guarantee stronger vendor requirements for change related SLAs, thus enabling us to realize technical changes more quickly. - Customers: Our general terms and conditions that we apply for our PKI based business allow for actions to ba taken due to security incidents. Of course, general contract management and the related change management always relies on both sides of the contract to be able to change the content of contracts quickly and we have seen very different ways in which customers operate in this context. It is therefore our standard contractual practice to always add technical specifications as an appendix to the main contract in order to facilitate a much speedier update of this type of documents. -------------------------------------- For example, did you consult with your auditor regarding the DNqualifier, and did they advise you it would be compliant? If so, that's useful information to share. If not, that seems a useful addition to your process as changes come through the CA/Browser Forum. These are the types of holistic mitigations and understanding we're attempting here. ANSWER: We are in constant consultation with our auditor and of course the observations that are the basis of this discussion will be dealt with in the next audit report (our next main audit is scheduled for October this year). The expertise which lead our decision was based on the following academic input: “It is required by CA/B forum to include at least 64 entropy in the serial number of a certificate. Due to the prefix property of the usual hash functions the hash values of any data a|x1|x2|b|c and a|y1|y2|b|c are the same if the hash values H(a|x1|x2) and H(a|y1|y2) already collide. Such a collision is build up by an attacker modifying the blocks (x1|x2) and (y1|y2) knowing the block a. The colliding certificates attack is based of two colliding data, a good data and an evil one. The good data will be certified and due to the collision the CA signature is good also for the evil certificate. The necessary condition for this attack is always the complete knowledge of the good data (up to the collision) to be certified. Integrating an unpredictable value in the first block prevents this attack, because the following blocks cannot be adjusted ahead. In the considered case the dnQualifier carries additional randomness. The unpredictable data is distributed in this case over the serial number (24 bits), the dnQualifier (32 bits) and also the validity time. Because this certification approach was only made for asynchronous production processes it seems to be reasonable to consider also an entropy of the notBefore date. Due the procedure of processing these requests it is estimated as at least 10 bits (1000 seconds = 16 minutes). Therefore we have the required 64 bits of randomness. Additionally may be said, that collision attacks work today only for the SHA-1 hash or weaker functions. But the hash function used here is SHA-256, which is resistant against this kind of attacks.” - Source: Prof. Dr. Ernst-Günter Giessmann, Humboldt University of Berlin ------------------------------- Can you confirm your update is scheduled for this Wednesday? Just wanting to follow-up and make sure timelines are met :) ANSWER: Yes – as scheduled on August, 24th 2017 BTW: all of our affected customers are already informed. Certificate replacement has been started. Can you expand this timeline more, beyond "by the end of the year"? ANSWER: Yes. Our release road map schedules this feature for December, 21st 2017. ---------------------- If we forgot any open issue please do not hesitate to come up to us. Dr. Martin Riegel, D-TRUST GmbH Arno Fiedler

Ryan Sleevi

Comment 5

•

8 years ago

Attempting to summarize all the information provided so far: 1) dNSName containing '/' - See Comment #0, Comment #1, Comment #4 - Existing Mitigations: - Certificates require two independent parties to approve ("four eyes principle") - Remediation: - 2017-08-24 - Hotfix to systems to validate CSR against RFC 5280 - 2017-08-24 - Hotfix to update validation software UI to reduce risk of mistakes - XXXX-XX-XX - Improved software testing to consider such cases 2) Serial Numbers with Insufficient Entropy - See Comment #0, Comment #1, Comment #4 - Timeline: - 2016-09-30 - BRs require "CAs SHALL generate non‐sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG" - 2017-05-15 - D-Trust implements the necessary change - 2017-07-XX - D-Trust realizes that one customer platform for OV certificates was not updated - 2017-07-07 - D-Trust disables issuance for this customer platform - Existing Mitigations: - Using the DNqualifier to include additional entropy (previously industry accepted, this was rejected by vendors as an appropriate source of entropy) - Remediations: - 2017-08-08 - Updated CA POCs to ensure more reliable Browser<->CA Communications - 2017-08-25 - In discussion with vendors to ensure stronger vendor requirements for change-related SLAs - 2017-12-21 - New real-time dashboard for issued certificates to do more proactive auditing Is this a correct and accurate summary of the timeline of events and timeline for remediation? If so, I believe matters of concern are: - How was that interpretation about serial numbers made, when the Baseline Requirements are clear on what is technically required? How has that process been informed by this experience that such a determination was materially wrong and insufficient? - Was this proactively communicated with root stores regarding the 2016-09-30 delay - Was this proactively communicated with root stores regarding the 2017-07-XX internal audit determining non-compliance? - Do the controls proposed for Item 2 provide sufficient mitigation of the root cause? That is, what prevents D-Trust from interpreting some other specific technical requirement and choosing to do something similar-but-different, as a migration path? - Systemically, is D-Trust capable of enacting timely changes to the issuance system, given the overhead related to both contracts and software deployment. I haven't seen answers in either the timeline or the remediation plan that would address these concerns.

Arno Fiedler

Assignee

Comment 6

•

8 years ago

Hello Ryan, we hope that we answer your open questions effectual: ---------------------------- Issue 1) dNSName containing '/' - See Comment #0, Comment #1, Comment #4 - Existing Mitigations: Certificates require two independent parties to approve ("four eyes principle") - Remediation: - 2017-08-15 - The Certificate was revoked - 2017-08-24 - Hotfix to systems to validate CSR against RFC 5280 - 2017-08-24 - Hotfix to update validation software UI to reduce risk of mistakes - 2018-03-31 - Improved software testing to consider such cases Issue 2) Serial Numbers with Insufficient Entropy - See Comment #0, Comment #1, Comment #4 - Timeline: - 2012-07-01 BRs require “CAs SHOULD generate non-sequential Certificate serial numbers that exhibit at least 20 bits of entropy.” - 2012-11-15 D-Trust started using the dnQualifier instead of longer serial numbers, since certificate issuing system wasn’t capable to generate these with that length. - 2012-12-13 - Public discussion within the Mozilla Forum and recommended for Root inclusions ended - 2012/ 2013 - All demo certificates contained the dnQualifier and were accepted as compliant during every single root inclusion process we passed in these years. - 2016-09-30 - BRs require "CAs SHALL generate non‐sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG" - 2016-10-27 - D-Trust sent e-mail to browsers about delayed implementation - 2016-12-01 - D-Trust implements the necessary change for the enterprise platform - 2017-05-15 - D-Trust implements the necessary change for the retail platform - 2017-07-05 - D-Trust realizes that one retail customer platform for OV certificates was not updated - 2017-07-07 - D-Trust disables issuance for this customer platform - Existing Mitigations: Using the DNqualifier to include additional entropy Ryan: (previously industry accepted, this was rejected by vendors as an appropriate source of entropy) ANSWER: We never received this rejection neither by e-mail nor by the forum. We still expected to fulfill entropy requirements, even the entropy has not been set to the serial number (also see Comment #4). - Remediations: - 2017-08-08 - Updated CA POCs to ensure more reliable Browser<->CA Communications - 2017-08-25 - In discussion with vendors to ensure stronger vendor requirements for change-related SLAs - 2017-09-15 - D-Trust: Finalize process of revocation of all affected certificates - 2017-12-21 - D-Trust: New real-time dashboard for issued certificates to do more proactive auditing -------------------- Ryan: Is this a correct and accurate summary of the timeline of events and timeline for remediation? ANSWER: Yes, we added some missing details. Ryan: - Was this proactively communicated with root stores regarding the 2016-09-30 Delay ANSWER: See updated time line. We communicated on 2016-10-27. Ryan: - Was this proactively communicated with root stores regarding the 2017-07-05 internal audit determining non-compliance? ANSWER: Mistakenly this internal audit result was not communicated to the root stores immediately. Ryan: - Do the controls proposed for Item 2 provide sufficient mitigation of the root cause? That is, what prevents D-Trust from interpreting some other specific technical requirement and choosing to do something similar-but-different, as a migration path? ANSWER: Our multiple mentioned internal security measures as well as the extensive auditing of our systems and processes do reduce the risk of deviation from standards to the smallest possible extent. Ryan: - Systemically, is D-Trust capable of enacting timely changes to the issuance system, given the overhead related to both contracts and software deployment. ANSWER: We strongly believe that we are capable to do this. -------------------------------------------------- Best regards and enjoy the weekend. Dr. Martin Riegel, D-TRUST GmbH Arno Fiedler

Ryan Sleevi

Comment 7

•

8 years ago

I can confirm that, on 2016.10.27, multiple root store operators were notified of the delay, with anticipated resolution of 2016.12. I can confirm that, on 2016.12.12, multiple root store operators were notified that D-Trust had successfully implemented the controls. With respect to "previously industry accepted", this can be seen through the changes to the Microsoft Trusted Root Program technical requirements. - For example, Version 1.1 lists the specific technical requirements - https://social.technet.microsoft.com/wiki/contents/articles/20899.windows-root-certificate-program-technical-requirements-version-1-1.aspx removed this accepted language in Version 2.0 ( https://web.archive.org/web/20131116115134/http://social.technet.microsoft.com/wiki/contents/articles/1760.windows-root-certificate-program-technical-requirements-version-2-0.aspx ) (search for "For all end-entity certificates, the contents of the serialNumber field" to read the new restrictions) Note that, as part of the timeline, it's worth noting - 2016.02.26 - Pre-ballot for serial entropy ( https://www.mail-archive.com/public@cabforum.org/msg00442.html ) proposed - 2016.07.08 - Ballot 164 is adopted - 2016.09.30 - Ballot 164 compliance required - 2016.12.12 - D-Trust meets Ballot 164 compliance During the discussions of Ballot 164, there was also discussion about the subject name approach. With that said, I believe this represents a viable remediation path, with a further emphasis that failures to inform of internal audit failures in the future may be seen as intentional, rather than process failures, and thus represent an important place to ensure reliable processes are in place to ensure prompt and timely communication (as is already contractually required for some root programs) Please provide an update on 2017.09.15 regarding your process in developing your revocation plan and your outlined timelines.

Whiteboard: [ca-compliance] → [ca-compliance] [remediation-accepted] Next Action: 2017-09-15

Arno Fiedler

Assignee

Comment 8

•

8 years ago

Hi Ryan, many thanks for your Statement. A detailed update will be provided on 2017.09.15, however this is a short update on the Status from our side. Kim Issue: Short serial numbers PostMortem: Referring to the BR requirements regarding the introduction of 64bit serial numbers, the timeline is as follows: 2016.02.26 Pre-ballot for serial entropy ( https://www.mailarchive.com/public@cabforum.org/msg00442.html ) proposed 2016.07.08 Ballot 164 is adopted 2016.09.30 Ballot 164 compliance required 2106.12.01 D-Trust meets Ballot 164 compliance 2016.10.27 multiple root store operators were notified by D-Tust of the delayed introduction of this feature, with anticipated resolution of 12-2016 2016.12.01 D-Trust notified implementation of the feature on the managed PKI platform 2017.05.15 D-Trust finalized implementation of necessary change for the retail platform 2017.07.05 D-Trust realizes that one retail customer platform for OV certificates was not updated 2017.07.07 D-Trust disables issuance for this customer platform Summary of issued certs: Issuing End Date EV: 2017.05.15: 63 EV TLS certs Issuing End Date OV: 2017.07.07: 607 OV TLS certs A detailed survey of the affected certs was provided. Mitigation/Remidiation: As we were not able to meet the 2016-09 deadline due to dependencies on necessary changes in 3rd party software, we decided to add additional entropy in a separate certificate field. We ran this past several cryptographic experts and received a positive feedback as to the suitability of this measure. Also during the discussions of Ballot 164, there was a discussion about the subject name approach, i.e. our chosen remediation path. Furthermore it should be noted that the certs are using SHA-256 as per BR and that currently there is no known collision attack for SHA-256. Therefore we fully acknowledge that we were not able to meet the BR to its full extent at the 2016.09 deadline but are also convinced that we provided ample remediation. The necessary changes in order to achieve formal compliance with BRs for the managed PKI platform were implemented in 2016.12, while the changes for the retail platform were applied a later stage. The fact that no communication regarding this as well as the results of the internal audit on 2017.07.05 occurred is a regrettable mistake on our side. We acknowledge that a prompt and timely communication is a mandatory task that we will aim to comply with to the full extend in the future. To this end we are amongst other measures deploying additional internal resources to deal explicitly with these topics? In order to enhance the quality assurance during issuance we are setting up both manual random checks as well as automated compliance checks in our issuance system. We are also planning to log both EV and OV certs to CT in the future. Revocation Plan: No immediate revocation was scheduled as we consider this incident not to be a critical security issue (notwithstanding the fact that the issuance did not adhere to the exact requirements of the BR requirements in the period 2016.09.30 – 2016.12.01). All customers were contacted and were required to exchange certs as soon as possible, BR compliant replacement certs were provided to all customers. A considerable number of certs have already been replaced and will be revoked in due course. However some costumers with larger deployment stated the fact that an immediate replacement of certs of their deployment would potentially be very harmful to their respective users, as many of these rely on techniques like certificate pinning for enhanced security. Therefore these customers will replace certificates in sequential phases and the corresponding certs will be revoked by us once we receive signal from the customers that the respective phase was completed successfully. A concise summary on the current status will be provided by 2017.09.15.

Kim Nguyen

Comment 9

•

8 years ago

Final Report on Incident ** dNSName containing '/' https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ PostMortem: An incident was triggered by a bug in mozilla.dev.security.policy 07-08-2017. Issuance was stopped immediately at 07-08-2017 Analysis yielded the following results: Validation is based on both a four-eyed principle “human” approach as well as a tool based automated validation. The GUI of our validation software backend which our team is using had some usability and visualization related issues. This implied that the way multiple SANs were displayed had potential for mistakes. We released the improvement of the backend GUI on the 2017-08-24 as previously announced. The bug mentioned with respect to the CSR Validator was that the CSR validator didn’t filter prohibited characters correctly and was introduced by the previous release but was not recognized during test. Mitigation/Remediation: Existing Mitigations: Certificates require two independent parties to approve ("four eyes principle") Remediation: 2017-08-15 - The Certificate was revoked 2017-08-24 - Hotfix to systems to validate CSR against RFC 5280 2017-08-24 - Hotfix to update validation software UI to reduce risk of mistakes 2018-03-31 - Improved software testing to consider such cases In order to enhance the quality assurance during issuance we are setting up both manual random checks as well as automated compliance checks in our issuance system. Also a case-related awareness training was performed. Revocation plan: The cert was revoked, a new BR compliant cert was issued for the costumer.

Flags: needinfo?(kim.nguyen)

Kim Nguyen

Comment 10

•

8 years ago

Final Report on Incident ** Short / sequential-looking serial numbers https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ RESOLUTION: https://groups.google.com/d/msg/mozilla.dev.security.policy/UnR98QjWQQs/O-Hf5T4WBwAJ Incident Report Issue 2: Short serial numbers D-Trust GmbH, 2017.09.15 PostMortem: Referring to the BR requirements regarding the introduction of 64bit serial numbers, the timeline is as follows: 2016.02.26 Pre-ballot for serial entropy ( https://www.mailarchive.com/public@cabforum.org/msg00442.html ) proposed 2016.07.08 Ballot 164 is adopted 2016.09.30 Ballot 164 compliance required 2106.12.01 D-Trust meets Ballot 164 compliance 2016.10.27 multiple root store operators were notified by D-Trust of the delayed introduction of this feature, with anticipated resolution of 12-2016 2016.12.01 D-Trust notified implementation of the feature on the managed PKI platform 2017.05.15 D-Trust finalized implementation of necessary change for the retail platform 2017.07.05 D-Trust realizes that one retail customer platform for OV certificates was not updated 2017.07.07 D-Trust disables issuance for this customer platform Summary of issued certs: Issuing End Date EV: 2017.05.15: 63 EV TLS certs Issuing End Date OV: 2017.07.07: 607 OV TLS certs A detailed survey of the affected certs was provided. Mitigation/Remidiation: As we were not able to meet the 2016-09 deadline due to dependencies on necessary changes in 3rd party software, we decided to add additional entropy in a separate certificate field. We ran this past several cryptographic experts and received a positive feedback as to the suitability of this measure. Also during the discussions of Ballot 164, there was a discussion about the subject name approach, i.e. our chosen remediation path. Furthermore it should be noted that the certs are using SHA-256 as per BR and that currently there is no known collision attack for SHA-256. Therefore we fully acknowledge that we were not able to meet the BR to its full extent at the 2016.09 deadline but are also convinced that we provided ample remediation. The necessary changes in order to achieve formal compliance with BRs for the managed PKI platform were implemented in 2016.12, while the changes for the retail platform were applied a later stage. The fact that no communication regarding this as well as the results of the internal audit on 2017.07.05 occurred is a regrettable mistake on our side. We acknowledge that a prompt and timely communication is a mandatory task that we will aim to comply with to the full extend in the future. To this end we are amongst other measures deploying additional internal resources to deal explicitly and exclusively with these topics. In order to enhance the quality assurance during issuance we are setting up both manual random checks as well as automated compliance checks in our issuance system. We are also planning to log both EV and OV certs to CT in the future. Revocation Plan: No immediate revocation was scheduled as we consider this incident not to be a critical security issue (notwithstanding the fact that the issuance did not adhere to the exact requirements of the BR requirements in the period 2016.09.30 – 2016.12.01). All customers were contacted and were required to exchange certs as soon as possible, BR compliant replacement certs were provided to all customers. A considerable number of certs have already been replaced (see below for details). However some costumers with larger deployment stated the fact that an immediate replacement of certs of their deployment would potentially be very harmful to their respective users, as many of these rely on techniques like certificate pinning for enhanced security. Therefore these customers will replace certificates in sequential phases and the corresponding certs will be revoked by us once we receive signal from the customers that the respective phase was completed successfully. Of the in total 670 affected certificates: - 266 have already been replaced by customers and were revoked on 2017.09.15 - 20 are in the process of being replaced and will be revoked on 2017.09.29 - The remaining certificates are associated with several customers with large deployments as mentioned above and will be revoked in several steps starting already this week, followed by several consecutive phases of exchange. There are a few especially sensitive certificates for which an exchange would imply a considerable impact on the continuity of customer services and therefore will be exchanged within their normal lifecycle (i.e. in the second quarter of 2018). We will inform the public on the process of exchange and revocation in regular intervals. Kim Nguyen, 2017.09.15

Ryan Sleevi

Updated

•

8 years ago

Whiteboard: [ca-compliance] [remediation-accepted] Next Action: 2017-09-15 → [ca-compliance] [remediation-accepted] Next Action: 2017-09-29

Kim Nguyen

Comment 11

•

8 years ago

Update on Incident ** Short / sequential-looking serial numbers https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ RESOLUTION: https://groups.google.com/d/msg/mozilla.dev.security.policy/UnR98QjWQQs/O-Hf5T4WBwAJ The mentioned 20 certificates were replaced and accordingly revoked an 2017.09.29 The exchange in large deployments is ongoing and we are in close contact with these customers to Monitor this. Next update will be provided on Friday, 2017.11.03. Kim Nguyen, 2017.08.29

Gervase Markham [:gerv]

Comment 12

•

8 years ago

(In reply to Kim Nguyen from comment #10) > 2016.10.27 multiple root store operators were notified by D-Trust of the > delayed introduction of this feature, with anticipated resolution of 12-2016 It is true that this happened, and Mozilla did not object at the time. Therefore, we do not require expedited revocation of certificates with short serial numbers issued before that date. Any after that date should be revoked very quickly. > There are a few especially sensitive certificates for which an exchange would imply a considerable impact on the > continuity of customer services and therefore will be exchanged within their normal lifecycle (i.e. in the second > quarter of 2018). This sounds like a very bad position for your customers to be in; if they had a key compromise, it would either significantly affect their business, or make their communications insecure. You should impress upon them the need for certificate agility to be possible. Gerv

Ryan Sleevi

Updated

•

8 years ago

Whiteboard: [ca-compliance] [remediation-accepted] Next Action: 2017-09-29 → [ca-compliance] [remediation-accepted] Next Action: 2017-11-03

Kim Nguyen

Comment 13

•

8 years ago

Update on Incident ** Short / sequential-looking serial numbers https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ RESOLUTION: https://groups.google.com/d/msg/mozilla.dev.security.policy/UnR98QjWQQs/O-Hf5T4WBwAJ The Exchange in large deployments is still ongoing. To address Gervs concerns as expressed in comment #12, our customers are quite aware that in case of a key compromise or a similar incident, immediate revocation (i.e. within 24 hours) would be necessary and the customers are able to perform this, but only at the cost of severe service interruptions due to certificate pinning etc. But they are certainly able to deal with this topic. Furthermore all the remaining certificates where issued within the period of 10-2016 to 12-2016. All other certificates issued after that date were revoked and replaced by 2017.09.29 Next update will be provided on Friday, 2017.12.15 Kim Nguyen. D-Trust, 2017.11.03

Kim Nguyen

Comment 14

•

8 years ago

Of the affected 670 certificates in total 286 were replaced by 2017.09.29 (see above in comment #10). The exchange in large deployments is still ongoing. Here additional 123 certificates were replaced by 2017.12.15. Next update will be provided by 2018.02.16. Kim Nguyen. D-Trust, 2017.12.18

Wayne Thayer

Updated

•

8 years ago

Whiteboard: [ca-compliance] [remediation-accepted] Next Action: 2017-11-03 → [ca-compliance] [remediation-accepted] Next Action: 2018-02-16

Firefox Bug Husbandry Bot

Comment 15

•

8 years ago

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

QA Contact: gerv → wthayer

Enrico Entschew

Comment 16

•

8 years ago

The exchange in large deployments is still ongoing. 261 certificates are still affected. Next update will be provided by 2018.06.15. Enrico Entschew, D-TRUST, 2018.02.15

Wayne Thayer

Updated

•

8 years ago

Whiteboard: [ca-compliance] [remediation-accepted] Next Action: 2018-02-16 → [ca-compliance] [remediation-accepted] Next Action: 2018-06-16

Ryan Sleevi

Comment 17

•

8 years ago

Wayne: Does this seem like an overly long remediation period? Based on Comment #10, this seems to be suggesting that it takes over a year to replace ~600 certificates. From an ecosystem perspective, it seems deeply unhealthy to support that notion. Based on Gerv's https://bugzilla.mozilla.org/show_bug.cgi?id=1390990#c12 , it also seemed like Gerv thought this was long. Basically, zero progress has been made since Comment #14 - 670 - 286 (2017.09.29) - 123 (2017.12.15) = 261

Flags: needinfo?(wthayer)

Jonathan Rudenberg

Comment 18

•

8 years ago

I don't think this remediation timeline is reasonable. This bug was filed on 2017-08-16, and the original message disclosing the issue was sent on 2017-07-18. Hundreds of misissued certificates have still not been revoked. Additionally, the proposed next update date is in four months, bringing the total amount of time since the issue was disclosed to 11 months.

Wayne Thayer

Comment 19

•

8 years ago

(In reply to Ryan Sleevi from comment #17) > Wayne: Does this seem like an overly long remediation period? > Yes, and I was surprised that the response to Comment #10 wasn't stronger. > Based on Comment #10, this seems to be suggesting that it takes over a year > to replace ~600 certificates. From an ecosystem perspective, it seems deeply > unhealthy to support that notion. Based on Gerv's > https://bugzilla.mozilla.org/show_bug.cgi?id=1390990#c12 , it also seemed > like Gerv thought this was long. > My pessimistic read of Comment #10 is that D-Trust plans to let these certificates naturally expire. I presume this means that the process could take until mid-2020! > Basically, zero progress has been made since Comment #14 - 670 - 286 > (2017.09.29) - 123 (2017.12.15) = 261 I hadn't noticed that. Going back to Comment #10: >There are a few especially sensitive certificates for which an exchange would imply a considerable impact on the continuity of customer services and therefore will be exchanged within their normal lifecycle (i.e. in the second quarter of 2018) 261 remaining certificates certainly doesn't meet my definition of "a few". I would like to see all of these certificates replaced by 30-June 2018. That is still an extraordinarily long time from when this was first reported, but seems tolerable given today's date and their prior communication of "the second quarter of 2018". Given that D-Trust has chosen to violate the BRs so clearly and for so long rather than cause any inconvenience to their customers, I also expect to see this reported as a qualification on their next audit.

Ryan Sleevi

Comment 20

•

8 years ago

(In reply to Wayne Thayer [:wayne] from comment #19) > >There are a few especially sensitive certificates for which an exchange would imply a considerable impact on the continuity of customer services and therefore will be exchanged within their normal lifecycle (i.e. in the second quarter of 2018) > > 261 remaining certificates certainly doesn't meet my definition of "a few". > I would like to see all of these certificates replaced by 30-June 2018. That > is still an extraordinarily long time from when this was first reported, but > seems tolerable given today's date and their prior communication of "the > second quarter of 2018". It's not clear that Comment #12 found that reasonable. Certainly, there were constraints/expectations placed on that transition. For example, it's unclear whether all 261 remaining were issued prior to 12-2016.

Enrico Entschew

Comment 21

•

8 years ago

All 261 certificates were issued prior 1st Dec 2016. We are in intensive contact with our customers. Due to the fact, that the certificates are used in a complex ecosystem the exchange takes longer than expected. As soon as we get new information we will share them here.

Wayne Thayer

Comment 22

•

8 years ago

Enrico: will all 261 remaining certificates be revoked by 30-June 2018? I am asking you to set a deadline with your customers rather than allowing them to replace the certificates whenever they find it convenient. Also, will you attach the list of the remaining 261 unrevoked certificates to this bug?

Flags: needinfo?(wthayer) → needinfo?(enrico.entschew)

Enrico Entschew

Comment 23

•

8 years ago

Hi Wayne, Towards our customers we communicated the strong wish of Mozilla to revoke all certificates with short serialnumbers until June 30, 2018. As mentioned before this is a really challenging situation with the certificates being used in complex and sensitive infrastructures. We are still working on a solution and will keep you updated. Please find attached the Fingerpints (SHA-1) of all certificates with short serial numbers that are still valid. Best regards Enrico 1 5811ebb0dd640f5d8a81770a0f73badfe54bf495 2 9391f3453730c0878e39021fb42fd710bbb58596 3 57bdc88da9eca6f9b64c92de0986aa2f4724063f 4 4a22f82472ac8009ccbeac58802b504835778305 5 a4b53a346b34227bbef5c48dbb959c540e77051e 6 19844ec2066c1e0db833c426c75134682858cf7c 7 7f65ad201da8c34a49ace41f652d96b0522ae367 8 cf3eaeb6ab749e0298ced21664d59f9a52c6ff22 9 34bb48ecd61cf5fd79ed7c1dfe777e6896f7d00c 10 388dbdee423a7eeb88806c423e721764c84fdcb5 11 3e9ff702e414712ef9ec69465c2a73a70e760bb1 12 a467930c7a0175b173a3c74abdad42911bb168a9 13 c2c5844b093e20d24227cb7cda51c60453a3a200 14 c0e3d518ce32bc6fc10ec54844344cb56e18f67f 15 3ed2b8ee5569ef38a530bf9d56314fb5484fb3e8 16 5df34bf416b1184b42fb1536e8e868555aaad665 17 1b22eca6220e861561af5089b18b6a5ef272279b 18 1fa3b56cfa7a91252e3dd2e211eacfc0215675ee 19 64d7f15de6625ca9eb3fb653a96a7a0b20eaf8fe 20 81b95525337f010dd52f68ee00f9bd7cf13ffb0a 21 475cb9d580b5694cd4b54e5c984dbab0f2565655 22 3a064c8bef28a87d17b560bab4393ab7239a4e6b 23 0fdc73750268ce2031edcd4d31d27de2d56bcac1 24 e2b290014d0ef3250b75940323c336c9487efd00 25 7a3bed1bcb925900b6cef3d59bd3332686142d11 26 cd3629139aaa6584820eeee09e1281c06062ebb3 27 7836cc5060d9268eacbab743e8c5be066451dd92 28 cd21bdecdd4225c0fd059035fd17c3f584785c33 29 f76f8aab2283a6f5491182c9212f254134101d78 30 9b73b43acfcba73480ea499a66d12459ca22ca71 31 2f20eb474a43fd3ba12beed31425810d1b6cafb3 32 0d573f3ffec336880a13229b3c31f514fd7031b1 33 640be4c22f233051d83ca3f6d345904aab8ac21d 34 0b7c0bf39eb5d45917082aa9e859db3080edc789 35 be136dc11bd5655517756cde13a80d181c667a66 36 0a58a9354352b03286bef732f7234f0fa1a7adb4 37 b1e97b10253ff623073db399fd880e02df370ab2 38 2183fb8bf4d6607a08da2ccf1bd29ff285fb8bbf 39 20424a566cf93761bb7ee3b4255fc8ca8d517d58 40 1e13af2cf6645c5b3ac151ec4031ac7bfa612fc3 41 ab9643a1b8dcd547f35de0c57554e415e47401a4 42 e9b55289d8a73dfbd8f2931e47f3897bbdd283bf 43 14bc458c4b87eb572a073413e68022983c40e6cc 44 9ca26fc1f8422a5da567dcec622761362fd2b1d4 45 2a727f73ae355baab1309462e5690233eed4bfb1 46 e6ae6c49fe0408f3a0cb94f29a9f6bd43b96a937 47 1270365092575e1ca8ea9d44ec234b00fee651fb 48 8bc47380c1e08a3cb6ae35f6ca17933a7b8049d0 49 a03d9c6178342eee9b0537a3de8e83171df9c926 50 6ec3e4556f95d207c63353df46c1782c23d200f1 51 49317e3c0fd994b0d9b7fa93515b7e78c5f9b45a 52 893008e9df33166b8ebd7e2efd1391676f5d581c 53 7a0da2b74dd32c82f62a8c735446a51020c3983f 54 546cd375077ccfc657f3d97adedc6a23e46d46de 55 d43bd3bfdd1a35423055f7e3d625341fb329d7cb 56 c270bb7975e724b0f4fd365d2cb3ff8c9f43a61d 57 662737a17dd21c9b90251cc8b069a0aa70e1fbab 58 bf865019266d48dfd2d9f701025a146692974f1d 59 d1d7643235cfc827e0f3f0d68cd85af89049c1bb 60 238da5114e7978a4717f8afd6747ef511a841d4e 61 6a1c081d147754e772d0b95c33a1114c00cea167 62 e55dad9a832a17fe74fb0891cb27077277227e80 63 18301362933bf8dc8818293c914e7eb635376b18 64 ebe6e7feee92e5fc0ef23ae17b7f905b86dd5a2b 65 f5287a49e127cfcbb5d4395188ede56df2bdd011 66 09487c38979fc0df327378e7d4eeb7d1ecb0b2c9 67 44bf0c7307a7ba30558f9028df46bff426b2499d 68 a9d918047a75de253f6173d96032b6c206ad3023 69 47bc3660e9d43b62aaac8ab03dd93fc83c13fab2 70 f2937dd2f3b35d34d5f09bcdcf860a285de0723e 71 0428de85c6c7fec7b10e1437194ca000d2c083af 72 416b3c0f65e0eacd204e9955e31fbab91ced1bc5 73 4925aa43776a06694fbc91f4e55a089d79aa6bbe 74 f6720ae62ec3bab6ab58c58647250aae333da602 75 145e19b8e886547829e9163f054eb495bb99d15f 76 808d4d54aad5a40d182d61789fef55fd72d48b80 77 0d18b20360ef75dc9ea026f7516affc49e4e4f26 78 157232a6e5978051a37ba30e1136e633d315ddf2 79 a217d2cd75f23cbeb6b2e086ed350d65701c8b48 80 8232c0302de94e475cb57ff606f6aefd2ee27ab7 81 f44e3cb4c33bf1945fd24e67d524014117807670 82 66b6802e63363c830db3b19128ad173a844b08c5 83 81a028746939f92cec836d554ce1127d8e893a34 84 c49e987d939b0caa80f20036939df270a517f3ae 85 c6e4569556e55bd930a40706936a6dc12c6bf5d8 86 375174cfe3196077f28488124fee0e43fa8e6b03 87 f645ace091227a5da6a10706fcf86cbf4bbc990c 88 fed89fad483804fbf1eaa607cf5861ad0c3e6a3f 89 1b2716bde7003a86c054627fdd3659484f299047 90 08da2adcf80d0efaba5e377f4014aa9700d6d410 91 f569d3205386c9805c90fd66bb6587153d33bf13 92 9b6884fe77ea39e4b7580792189e614b0b7085fe 93 a4777c56810323a8c848a1cb0242f5a394d06cfc 94 794c07fb3b32e2978296c8841523c3222bc3d82a 95 6bb17aa29eba65f43cbc5d43452a839c1ecb2b54 96 707b91a4645c34e021db46d50a01c4e396128f7f 97 801493f4205b117f6f0fc627309e4ca1e3bb6233 98 115f2b25691f5c3ea583d38600adf90f0aa0636d 99 4d30447678df0af61743e7fe3e22d108f8349502 100 e29e567d13e0a76a78babf49a0b5386baaeb3e20 101 77144ec72c3f4419d85eee974cc86d6758bc900a 102 6b59ddacb80ff9769bba9f01640026edaa9d989a 103 15b44d840fe27e60dd293cd2e8d898a6426dc013 104 bb5dab0e56b000bf27bfe462fcb7f136162a83b0 105 587429adf8f908c680beb1f745a6131543368d64 106 6420ab00a8ddd4b62eb30aae656ec65b7b7044c7 107 2b8aefab47d710f702fa27b57728142cf98b4848 108 b2403e422d5a5d7d9d670d4cda9870da8de5e74a 109 46f4528660fcdbc99b62d7ab34bec64b5c1f205f 110 36c98b69a929706d675e98e208766adefaf0a2e0 111 f99d59bf368653d7203aa77f4932ee269335d916 112 ccec3a0b02b943f02f7956a1912495f0d5874c76 113 89739c73d0f4ce7497425d037e912086eaa23294 114 8be1d9478e4c316a385ad9f8a049e4b7fc1b6a13 115 00b89a0ccc1e67089f32e31feb3420159ae1b890 116 5c25250d92fc43b371c429617366111fc7b83421 117 477ad2deea049fba1ec185a6bd73cb4fb4bca99e 118 45f48c5d80794315e0b03d159350b81c2c1e67e3 119 a212986aec94b28372bfce17c66ea6096a5b9f40 120 fb5c4e8c4d15ee546462450a034f72e4ff2e80a6 121 07237d3b131db432be4f61f5140336d479b959dd 122 030a66258548202da1408cbdd0ad1e0814ae73f5 123 46c7a2fe73f15d938955db97fb5688fc2c330579 124 d6f3c7a6ec21fbfa8deaabd3cf7bb57a527504a5 125 caa151878e7780929f5f91fcf925594e18278753 126 23f9dcb91482bd21df1beb0ed90e480f1e76122d 127 1fd83788863b1869c851d805bc0fa3fa42d41972 128 dbd4b3c10b16bd680406da076c3995c869feb214 129 5e95f435d07783865d42b60ac09dee56531f9257 130 2a0ec253a036a9e1ac8ee037b23392e02f09eec6 131 a51de1ed2836f6c6beff8a816640615a17d78347 132 009f08226d045f76bceb49655bc6975b2d89f6ee 133 68f9bb71e7b524fa707f43315a2d1ed949b8e667 134 b7296b1382dd6ba346a1ec3ea7f1601835f89700 135 37c7345b7b1930f9912bdf8084ff078cd551d44c 136 d60c2406e2fc4e31851620eebd412333a3952887 137 29f577bcffc2dbc9cb66db384b5ae246dd49ea57 138 85bf06cd89caa703be6b9d1fe5140f6fb4acfa00 139 91da775e3732d53136a8d4d67350d14147aa7987 140 1ce0317725e127cde25b1298e109d5d3a7eb612c 141 2847027dbb26972422805409d2e814faeb43287f 142 157056889dc06cbcd5ea7ff8e3e5e0ec9b61ae06 143 d013592c283308782f90e9c53d5f02e97658629c 144 65086e36850884ed8b6220b42c8299c8e84c7880 145 f751f63c79444596d580a2f67b31ea4b21668f05 146 80c15eaf3f9d84652d0c2d6bf66c2f3d1b3a801c 147 08d207955bcceae9f30cbf63e917fa64455ab0fd 148 c229d97e7944bc4299f2a3058a7abbf6bb04f764 149 20a57c700f5a20c26be9fc237f1cef7c986031c5 150 62efff2cc872ad4f133370defa3667b50a74df98 151 fd858d8acbd62fe1ca53af6e9818bc444c4b6af2 152 c5f92f63b487db41b57095a9f07c3580fa3a455e 153 faa858ca9e3c628cf0fd2d5bc607c5050ebb2644 154 0c27a19ee2c34d0a81cd7e446122561c325fbfc4 155 002e7a313971f2d8b1f4ed965e282c1e42de5b26 156 413019032805f07b0dde4dd58b3be53aade49d52 157 1af86cd62e860ef28775f2261fdb9aee82f32be4 158 d18296079f2b1a3de25d91890ed372cf3fadd16e 159 b992e1624e15363f0824067e8a3e48c26119a759 160 079411a5d7a3c0714c362914aa1ec4df6600fe16 161 ffa270f88fab082ec78b43c406398cd8da7c4ade 162 906c5e8271e5786a4cbbf4cc05e62fc950cfd45c 163 d6f01e56120320ced73701c62003d027a238cd74 164 9d3e50adcca17114143611815af4caca8c15429a 165 1b9969bbf3ce1ca4f19c69981c45420a8b027b74 166 9ead36244df6c01da8e369d62716486b59a2079d 167 f5445da081954b7cea2af07133b0f0ef8be7594e 168 bfb0e4a0ba81f47c72ddcfa18bd325d75062274e 169 95d9ba1956c1ec906e1180fc2f87af65da91f693 170 e3ba21554a3c27ad3b23c31aa84bf6bcff9ee498 171 6eeefd1d4d878b683e5b7045282204086201dfea 172 ebd15a0ca53a9e9b86015f5d7bad58bfdf508d9b 173 839bd0153cef24ce484443ceba02160fe09c768b 174 429acc7690c4fdb144ce3aa8fd27324aac14ab90 175 a8b5c6d8e1b7bdc7e8ebef0b20ed2dfd6c5660e6 176 f4b89cb4e430c216742d44a9a91bd83e2c502d68 177 f714a7a0a04cf05228b0f7f2524ab525de8fc90d 178 307de594bb2328adc296bb5b72beebe91a78c535 179 d55a791757ccd547415d829ed0575b99a884d580 180 d9f619edfdfb78eaeec9b728993a0d4202196aca 181 efd8a5d856d95c47746ba7f1b1e405bd91ccb6e9 182 c3a14ce679ae609d0ef066fb844e63bdaa5bdad6 183 d20ee391e538c112610f405a9462f40a9b7ff875 184 f0a9133613cbc186f1155739037d5e5d063d26b9 185 b85bb2c57cb7a7243769993e1c1710faaf91292c 186 14ad14a807cc31cdb0e92df54d32bdeeea5ce16c 187 ddd901e5d55b82e8f1966ca2957ec9add8b1cf8f 188 04c9f9fa354cf94b15dfb61f32781f041fbb5669 189 2abd5b029e08bf838a1c96864b8f4e91452af9c1 190 8ed71f358562f1d40da34d0c02f2d2ab73e2093b 191 074ebca354c3c36886b5e68cb398805ce8ed937a 192 1c42cca7a25691a5149247a8855365bb02a21ff1 193 d7cf54198963a0d94fae3fd5e1d3aa11ea9e81c1 194 b09c1bddb6f99a2c25103b76fc5473814a482bca 195 8c2a4760eb6da815e735df24e904979dcd6a61cb 196 ed6cd5cb6093cc48374f26a4276a75e4d1d36ac9 197 82e18b4443ae780d793fdf7134f1d30bd27538b9 198 0a241edfa828856e63aacc43f8e66f7e40d44884 199 b6d52abbe1981d3b94d11a08c54d13d575b5a9d7 200 c0ca999a8760a81e51c54edd43118b1ae1f2a522 201 de7fc60898d80ee174ae3d6d5d3f622f10d95c59 202 48fba98871e50110eb726d6752664d9ab8a22eb7 203 e1b9077be67bb2b11bd6c6045c6db201553bd0ea 204 8a6add48b62628023424a3abc57840ac896bbb86 205 7258a8bc8f50dedc1c242cdd412665862fe2781d 206 d2469d8d8711b78e93354c2696f7e7fc2c10eecd 207 958726dfc56411f96fc2dc199dca4ae3236fb4fd 208 220baa36c4e97f757367dc32246dd8f1ae6538d2 209 896eb671e678ad9fe3d27dc8192c4f59f24327b2 210 645968d8b0f26ef8f30394c9798cdf39ee87571f 211 47357174f7c874fc63635ce5b61c1c12975e2478 212 3347f0ec488e53ecdd4f1fb727b5f774344172c4 213 f3036898f4458f4528fe595306dd742204f4e072 214 53e1b8193c5ce875bb9cb703202bf2a227f414f6 215 be0280b276bfa044088e46cb7f7754eb1ebf3b80 216 3806e84841cf62d7d34a493dca19f4fc69a060a1 217 a118126d0d143b99f0f121143d7a8f2fec1518ff 218 4475be936b0e316143d496baa31257419d2b2dfd 219 c00e06853a29f32c029d1bfff5684a6aea26cf90 220 8c40d9b7121049e9bcb0babaab2a27c869533d27 221 32c660a35759fc2d57a24069318ff481171892a2 222 b0ff0ffefb48e5044386493d589ecbb1ac1e3f61 223 58b924ca97fac26e2808c3f02c525f56487f89a1 224 3220030e9b440b0bcf46f760a03ae196652d1b9b 225 ef0fba88967784fc95844dbb378db8f9d74b9fd2 226 eaf879e1bfeb03503b9cadfb5ae9758b6ed5c0c0 227 cd7510f2b574702306b12fcac6c5142cc692a25e 228 c48008c8e82c3de881ad180f333a7bf18ddab903 229 84e6f1e695babfe7147595e4c3a5dd1c814a2bef 230 ca332c6408e59f9a1065d319767220df6a45aaed 231 46c7eafde5c861bd2417b3dc1c8c01eb3665fca0 232 f4b801e7afcf412e511b2ac5acd167e3d3407640 233 eb619a5d5fa7a5cd7e958fb9b7ddc8102a987cf5 234 80d860b58e4f17fd12036df00844c1c9a7963ecb 235 7a62a3ff5fc5e1687e0d27a5f5c767af7f93e3aa 236 b18acdaf545d6a23441f9845dcea6c9e8b6e5c73 237 33c6cea46f2577a6616b7e85cde61f6914131239 238 d64b451703d95672dbac157a579fdd4ccfd61b88 239 baa0a7ef87c405e50ee6e6aec6b41bd472905da7 240 9edac93e9d4d02c91dc72b1b930433467fe031fb 241 8df8239aa0057cc19f2564ca9c7b7f69c46cff69 242 67f93f1c62b1863626091ce3b4f7cb66cc4450e2 243 b0d91c5574bcc38ab86693500b68dd993bc8a93c 244 a1df16fdd323436343b616cd7b5eadda01fc93ff 245 87823bc15f97a30fe7fc338e8a219368ab7ae080 246 adda41d9fa1362e88bcbeaacbe95cf98bb3a496d 247 5fbb619f13d5558d1daf90865d38bca1dec6ad70 248 ddf95fe3586434bc36d18051b3ee0b673003322c 249 4bfb9dd948cfea3fc8cd9b641742bebcb966ad7c 250 3e34133004a086a59aefbdd92298fd2312783a1d 251 54b8f64d0ab18152a0ce2895f13e454c18aeaaa3 252 5a0928c9d39abce837205595d23f071101378a99 253 f9bb56b7c1d4ccd812dad4a5c7d5abd6afe16aca 254 2ec7cf357bf377ddeae7353e380a629ee5d7458e 255 d075a5309941e7aa885fc897bfad992d3264fade 256 774951607a221ea8358b17f89daa96583f6ca6af 257 b41736fa63f7107cfcb151bf601238ff96abe1bf 258 cbc0b96bc1d4a9166bf38d1d177457f27a65bf47 259 1aeed84b0c073885c8a19ec08424d7e94a242500 260 1cb6d515cfef65c2ededac232e74b4b1d7e199ad 261 f70f91c928f60553f2b66796981f6a5e3cc1dbdd

Flags: needinfo?(enrico.entschew)

Jonathan Rudenberg

Comment 24

•

8 years ago

Hi Enrico, Would you please upload these certificates to one or more CT logs? Thanks!

Flags: needinfo?(enrico.entschew)

Enrico Entschew

Comment 25

•

7 years ago

Update on Incident: All certificates with short serial numbers are revoked. The revocation process was carried out in three phases. Of the in total 261 still affected certificates: - 1 was revoked on 2018.03.06, - 179 were revoked on 2018.07.02 and - 61 were revoked on 2018.07.16. Thus, no effected certificates with short serial numbers are longer valid. Regards, Enrico Enrico Entschew, D-TRUST, 2018.07.18

Flags: needinfo?(enrico.entschew)

Wayne Thayer

Comment 26

•

7 years ago

Enrico: thank you for the update. It's good to hear that all of these certificates have been revoked. Is it possible for you to publicly disclose all 261 certificates listed in comment 23 by publishing them to CT logs as requested in comment 24?

Flags: needinfo?(enrico.entschew)

Ryan Sleevi

Comment 27

•

7 years ago

Enrico: Do you have an update on this, 6 months later?

Wayne Thayer

Updated

•

7 years ago

Whiteboard: [ca-compliance] [remediation-accepted] Next Action: 2018-06-16 → [ca-compliance]

Ryan Sleevi

Comment 28

•

7 years ago

Enrico: Do you have any updates?

Kim Nguyen

Comment 29

•

7 years ago

Ryan: Enrico is out of Office right now, we will respond shortly with an update. Sorry for the delay! Kim

Enrico Entschew

Comment 30

•

7 years ago

Thank you for bringing this up. Sorry for the delay.

The actual status is that all effected certificates are revoked. We got the permission to publish some of the certificates to CT-logs, which will happen soon.

I will inform as soon as this is finished.

Flags: needinfo?(enrico.entschew)

Ryan Sleevi

Comment 31

•

7 years ago

Any updates?

Flags: needinfo?(enrico.entschew)

Enrico Entschew

Comment 32

•

7 years ago

The upload of the certificates is planned for next week.

Flags: needinfo?(enrico.entschew)

Ryan Sleevi

Comment 33

•

7 years ago

Enrico: Do you have an update?

Flags: needinfo?(enrico.entschew)

Enrico Entschew

Comment 34

•

7 years ago

We had the permission to publish 150 certificates to ct logs. The upload has been successfully completed on 2019/01/24 and 2019/01/25. The certificates are now public on 4 ct logs (https://ct1.digicert-ct.com/log, https://mammoth.ct.comodo.com, https://ct.googleapis.com/pilot, https://ct.googleapis.com/rocketeer). These are the references to crt.sh:

Flags: needinfo?(enrico.entschew)

Ryan Sleevi

Updated

•

7 years ago

Status: NEW → ASSIGNED

Ryan Sleevi

Comment 35

•

7 years ago

Wayne: Comment #34 slipped through the cracks in light of everything. That has a disclosure of 150 certificates, less than the 261 certificates requested in Comment #22. I'll punt this to you to decide whether to close, and/or whether further explanations and process mitigations are necessary by D-TRUST regarding future incidents (see Comment #18 / Comment #19)

Flags: needinfo?(wthayer)

Wayne Thayer

Comment 36

•

7 years ago

The response to this incident by D-TRUST has been deficient, especially with respect to the time it took to replace the defective certificates. It should be clear that Mozilla expects faster remediation in the future.

Enrico: Is there anything more that you can share to reassure us that D-TRUST's response will be better next time?

Flags: needinfo?(wthayer) → needinfo?(enrico.entschew)

Enrico Entschew

Comment 37

•

7 years ago

Wayne: First of all, I want to assure you that we regret this long period of revocation very much. We have learned our lessons. We adopted and implemented extensive measures to deal with the incident to prevent such delays in the future.

From our point of view, the facts of the case are as follows:
Unfortunately, the complete revocation of all certificates with short serial numbers issued after September 30, 2016 could only be completed by D-TRUST on July 16, 2018.
Most of the certificates were used in infrastructures, which customers described as complex and mission critical. Therefore, the revocation took place in several steps and in close coordination with the affected customers.

A) Revocation of certificates

From September 30, 2016 to May 15, 2017 63 EV TLS certificates and until July 07, 2017 607 OV TLS certificates with short serial numbers were issued. Of these, 409 certificates were revoked and replaced by December 31, 2017.

As of January 01, 2018, 261 certificates with short serial numbers had not yet been revoked. According to our customers, these were used in sensitive and highly complex infrastructures. We revoked the certificates in close cooperation with our customers until July 16, 2018.

B) Publication of certificates that were revoked in 2018

The hash values of all 261 certificates that were revoked in 2018 were published by us in order to create transparency. 150 of these certificates were published in CT logs. For 111 certificates, there is no customer approval for publication due to the contract. Therefore, no publication in a CT log has taken place here so far. We ask for your understanding for this situation.

What steps has D-TRUST taken to prevent such a delay in the revocation of defective certificates in the future?

As already mentioned, the entire incident has been extensively analyzed and measures have been taken to prevent such a delay in the revocation of defectively issued certificates in the future and to ensure that the certificates are revoked within the times specified by the BRs.

Expansion of the PKI team
The team to assess certificate incidents and respond immediately has been significantly expanded. In addition, the content of the training of the support team, especially on the subject of certificate incidents, was expanded and the team was trained accordingly.
Optimization of internal and external processes for the revocation of defective certificates
The internal and external processes for processing defective certificates and certificate incidents were revised and optimized. In addition, significant changes were made to communication on the website and in customer documents.
Contractual optimization/ revision of obligation to revoke and the customer's obligation to cooperate
The contract documents have been optimized in such a way that it is particularly clear that the customer has a duty to cooperate and that the CA has an obligation to revoke in accordance with the BRs in certain time windows.
Proactive customer communication regarding the CA's obligation to revoke and the associated risks for the customer's IT operations
We have again informed our existing customers about our obligation to revoke and drawn their attention to the risk to their IT operations if we have to revoke defective certificates within the revocation periods of the BRs. We have asked our customers to take precautionary measures.

With these measures, we want to ensure that the revocation periods in accordance with the BRs are adhered to in future.

Flags: needinfo?(enrico.entschew)

Wayne Thayer

Comment 38

•

7 years ago

Enrico: Thank you for this informative response. I now believe that all questions have been answered and this issue has been resolved.

Status: ASSIGNED → RESOLVED

Closed: 7 years ago

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Updated

•

3 years ago

Product: NSS → CA Program

David Lawrence [:dkl]

Updated

•

3 years ago

Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance] [ev-misissuance] [policy-failure]