Closed Bug 1716843 Opened 3 years ago Closed 2 years ago

E-Tugra: CA Certificate Missing from Audit Reports

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: bwilson, Assigned: dtokgoz)

Details

(Whiteboard: [ca-compliance] [audit-failure])

eTugra's audit report does not list the following CA Certificate:

E-Tuğra Nitelikli Elektronik Sertifika Hizmet Sağlayıcısı v2
SHA256 hash of certificate:
DD2CB2EEC52F5E96AB1CB8430952FB56C8E6ABDFF21EACD7684B1CD738AACCFF

https://crt.sh/?q=DD2CB2EEC52F5E96AB1CB8430952FB56C8E6ABDFF21EACD7684B1CD738AACCFF

They have indicated that the CA is only used to issue end entity certificates for digital signatures, but it appears that the CA is capable of issuing TLS server certificates.

Updating summary to match other bugs involving E-Tugra.

Summary: eTugra: CA Certificate Missing from Audit Reports → E-Tugra: CA Certificate Missing from Audit Reports

This CA is used to issue end entity certificates for digital signatures. It was created at March, 2013. This root will be included in scope of audit in next audits. The next audit is on end of this July.
We need day to propose an Indicent Report

This CA doesn't comply with the Baseline Requirements, though, and is supposed to be revoked. Including it in the scope of the next audit doesn't address the concerns.

Can you share whether you've reviewed past CA incidents regarding such omissions, and the expectations/clarifications?

Flags: needinfo?(dtokgoz)

Firstly let me correct that this intermediate CA used for Qualified Signatures according EIDAS standard and audited by Turkish ICT authority. During audits based on TLS and BR, this intermediate CA are being excluded by auditors.
Actually, in my first response, we thought that it is not problem to add this intermediate CA to next audit scope. After my first response, we start to detail discussion with our team and auditor about the case and including this intermediate CA in next audits, we saw that it is not sensible solution.

E-Tugra is a national TSP (Trusted Certificate Provider) in Turkey that was being operated from 2005. E-Tugra is audited twice in a year. First one is by a qualified auditors based on ETSI (EN 319-411-1) and BR, for TLS certificate. And second is by Turkish ICT Authority based on EIDAS Qualified Certificates Standards . We are in trusted TSP list of Turkish government (https://www.btk.gov.tr/elektronik-sertifika-hizmet-saglayicilari). All our CA Hierarchies are listed in https://e-tugra.com.tr/en/certificates-and-revocation-lists/

The aforementioned intermediate CA is belong to our 2nd Hierarchy and was issued on March 2013 at the same time with its root and other CAs (that are used for TLS certificates) in the presence of BR Qualified Auditors (due to EV requirements). (https://e-tugra.com.tr/en/certificates-and-revocation-lists/#2hierarchy)
This root was included in the list of Turkish TSP (http://depo.kamusm.gov.tr/depo/SertifikaDeposu.xml).
This root also included in trusted lists browsers for TLS certificates after completing inclusion process (https://bugzilla.mozilla.org/show_bug.cgi?id=877744) on May 2014.
Currently, there are hundred thousands of qualified certificates were issued from this intermediate CA until May 2020.

Due to limitation and restrictions of this model and changes on BR and related standards, we decided to change our model and to issue new roots separately according to usage purposes. In March 3rd, 2020, the new root CAs for only Qualified Certificates were issued and were included in Turkish trust list (http://depo.kamusm.gov.tr/depo/SertifikaDeposu.xml). We begun to use this hierarchy to issue qualified certificates in May 2020..
In March 18th, 2020 two new root CAs (RSA and ECC versions) were issued in the presence of ETSI Qualified Auditors for TLS Certificates. We started inclusion process with https://bugzilla.mozilla.org/show_bug.cgi?id=1628720 and related CCADB records. The new hierarchies are on https://e-tugra.com.tr/en/certificates-and-revocation-lists/#5hierarchy and https://e-tugra.com.tr/en/certificates-and-revocation-lists/#6hierarchy.

During ETSI Audits for TLS certificates, Auditors are testing all our CA hierarchies and our CA systems, and they reviews all our CA process, profiles and lifecycles are separated for qualified signature and TLS certificates. They prepare the audit reports with this observations. In our CA systems, due to limitation and system and security settings, it is impossible to issue a TLS certificates from this aforementioned intermediate CA. In May 2020, we stopped to issue new qualified certificates from this intermediate CA. This CA is used just creating CRL daily and we planned to continue till march 2023, its expiration date.

During our discussion inside e-tugra and with auditors, adding this intermediate CA in audits is not sensible solution, because it is never used for TLS certificates and this CA audited and monitored by Turkish ICT Authority regularly. Revoking this this CA is also not possible due to live qualified certificates and commitments to Turkish government. In this circumstance, we commit that no TLS certificate will be issued from this certificates as before.

Flags: needinfo?(dtokgoz)

Thanks, but this isn’t what I asked in Comment #3, nor does the plan outlined meet the longstanding requirements.

It sounds like you plan on intentionally not revoking this unconstrained sub-CA, which is capable of TLS issuance, and not appropriately audited. It appears you’ve not reviewed previous communications on the expectations, or the incidents. Is this correct?

Flags: needinfo?(dtokgoz)

E-Tugra reached out privately asking if I could clarify Comment #5. In the interest of keeping things transparent, I believe it's best to provide such transparency here, although it does raise concerns about E-Tugra's incident response.

It's worth noting that Comment #4 fails to provide the details requested in https://wiki.mozilla.org/CA/Responding_To_An_Incident , and thus does not seem to be the incident report promised in Comment #2.

The pattern of having unaudited sub-CAs, which this is, is a long-standing problematic practice that has been clarified for years. Here are a few incidents that E-Tugra should review:

This is not an exhaustive list, but this was conducted with 30 seconds of just scanning past incidents for audit issues. Comment #4 fails to demonstrate an understanding or awareness of these issues. The discussion about E-Tugra's intent, or its process, is irrelevant at this point: what matters is this is a CA that is technically capable of TLS issuance, and has been required for multiple years to be disclosed within the relevant audits.

E-Tugra's Comment #4 highlights that their auditor does not believe they can list "CA certificates not intended for TLS" within the scope of their TLS audit. However, E-Tugra has not demonstrated any review of the relevant past policy requirements or discussion that make it clear that this is explicitly expected, and has been for years. E-Tugra bears the responsibility for selecting an appropriate auditor that understands the requirements and expectations, and has failed to do so.

E-Tugra's Comment #4 suggests the resolution to this can be accomplished by a "commit that no TLS certificate will be issued from this certificate". However, such commitment is unacceptable: the only acceptable remediation for issues like this is to revoke the certificate, which is what is required of the BRs and Mozilla Policy. E-Tugra may decide to knowingly violate the BRs and Mozilla Policy, and not revoke, and if that's the case, the expectations are covered at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation . Mozilla may decide to add the CA to OneCRL to protect their users, but that does not mean other root programs will find the plan acceptable, nor does it mean that E-Tugra has not had a serious BR violation.

The request in Comment #3 was to ensure:

The requirements here have been clear and unambiguous for some time, E-Tugra has received multiple notices, and can see from the above issues list, that the only appropriate resolution for this issue is the revocation of the Sub-CA, which is a long-standing requirement of both the BRs and Mozilla Policy. If E-Tugra will not be revoking the Sub-CA, that's a new incident, and there are clear expectations for the information that needs to be provided.

Hi Ryan, the aim of my private email is to get more detail from you as a contributor of this bug to prevent some misunderstanding caused by my English and a little bit of help for finding the best solution. We started to review all similar cases and review policies again. Your last comment has made many positive contributions to our internal discussions and solution development.
The aim of the explanation in Comment-4 https://bugzilla.mozilla.org/show_bug.cgi?id=1716843#c4 is to give a brief of the case that we are in. We continue to discuss and try to find the best solution for that bug and try to prepare an Incident Report.
Our preliminary Incident Report as Follow

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.
    [Etugra]: We have received an email from Ben Wilson of Mozilla and an email from Bugzilla notifying us of this bug On June 16th 2021.

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    [Etugra]: On March 3rd, 2013, Creation of Root CA “E-Tugra Certification Authority” with validity 10 year.
    On March 16th, 2013, Creation of Sub CA “E-Tuğra Nitelikli Elektronik Sertifika Hizmet Sağlayıcısı v2” for qualified signature according to Turkish National Qualified Signature Law and Technical Requirements established on ETSI TS 102 042 that each year full audited by our auditors.
    In July, 2017 Mozilla CA Certificate Policy 2.4.1 was published and came into effect. In this policy clarifications were made around CA audits and requirements also were added for intermediates technically capable of issuing TLS certificates.
    In March, 2020 Separate root CAs were created for TLS certificates and Qualified Signatures Certificates.
    On May, 2020 Issuing qualified certificates from CA “E-Tuğra Nitelikli Elektronik Sertifika Hizmet Sağlayıcısı v2” was stopped.
    On June 16th, 2021: this bug was created.
    On June 18th, 2021: before starting an Incident Report, we shared our case and initial feedback to the public with comment-4.
    On June 18th, 2021 we started to investigate all cases, review all the policies, to analyze the root cause of the problem and find an acceptable solution.
    On June 21st, 2021 we started to communicate with Turkish ICT Authority and other third parties for create the best solution.
    On June 22nd, 2021 we began to study in detail of three solutions, that offered by Kathleen in many platforms such as https://groups.google.com/g/mozilla.dev.security.policy/c/M7NGwCh14DI/m/3V4Spe5ODgAJ. Provide a revised report for those covers all periods after 2017; Adding it to OneCRL (also with notifying other root stores for similar action); tring to create a revocation plan that is technically feasible and acceptable to the community within an applicable time frame.

  3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
    N/A.

  4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
    [Etugra]:
    Only A CAs in scope from the point of view of Mozilla and other Root Stores.
    https://crt.sh/?q=DD2CB2EEC52F5E96AB1CB8430952FB56C8E6ABDFF21EACD7684B1CD738AACCFF “E-Tuğra Nitelikli Elektronik Sertifika Hizmet Sağlayıcısı v2”.

  5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
    [Etugra]: See answer 4.

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    [Etugra]: E-Tugra had the similar logic as in cases (see Bug 1586125 and Bug 1605126), and came to the conclusion that there is a misunderstanding on management CAs that was not used in issuing TLS Certificates.
    The aforementioned intermediate CA was signed before the BRs came into effect. At creation time of that CA, technically constraining the CAs wasn’t a common practice, and as such all intermediate CAs belonging to “E-Tugra Certification Authority” were signed without any EKU or nameConstraints. When Mozilla sent a CA communication after publication of policy 2.4.1, E-Tugra answered that they were compliant with the Mozilla CA Certificate Policy.
    The usage of the aforementioned intermediate CA is only issuing certificates for qualified signatures. As indicated in our CP/CPS, certificates issued by this CA are only issued on QSCD’s just for usages in Turkey.
    This Aforementioned intermediate CA isn't used to issue TLS end-user certificates and are restricted in ETugra CA Management Systems and it is impossible to make it to issue a TLS with an error or with a human intervention. This CA is audited under ETSI Qualified Signature Standards every year. This thinking of ETugra caused that It is deemed to have “technical measures” to prevent issuance of TLS certificates.
    Due to the explanation above and in comment4, ETugra interpreted the expression “technically capable of issuing TLS certificates” differently than Mozilla aim.
    The reason that this problem hasn't been catched earlier was because of this assessment difference in logic built in ETugra thinking.

  7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
    [Etugra]: E-Tugra is determined to make the most appropriate solutions required by this bug as soon as possible.
    We are studying in detail three possible solutions that are offered by Kathleen (https://groups.google.com/g/mozilla.dev.security.policy/c/M7NGwCh14DI/m/3V4Spe5ODgAJ) . We will provide Updated Incident Report in a days.
    We believe that adding this CA to oneCRL is the right solution at the moment.

Flags: needinfo?(dtokgoz)

Thanks Davut. It's been two weeks since Comment #7, which both misses the expectations in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed and doesn't seem to be in a few days, as suggested.

It appears, at least, that the root cause being suggested here is:

The reason that this problem hasn't been catched earlier was because of this assessment difference in logic built in ETugra thinking.

Given the multiple communications, and, as you note, Kathleen and my 2019 remarks, it's unclear what can or should be done to improve E-Tugra's analysis and understanding of the compliance obligations.

This is equally concerning, since the Baseline Requirements and Mozilla Policy both include unambiguous language about "technically capable":
From BRs v.1.7.6, Section 8.1:

A Certificate is deemed as capable of being used to issue new certificates if it contains an X.509v3 basicConstraints extension, with the cA boolean set to true and is therefore by definition a Root CA Certificate or a Subordinate CA Certificate.

And from Mozilla Root Policy v2.7.1, Section 1.1

Intermediate certificates that are not considered to be technically capable will contain either:

  • an Extended Key Usage (EKU) extension which does not contain any of these KeyPurposeIds: anyExtendedKeyUsage, id-kp-serverAuth, id-kp-emailProtection; and/or:
  • name constraints which do not allow Subject Alternative Names (SANs) of any of the following types: dNSName, iPAddress, SRVName, rfc822Name

I highlight these not because I think you're suggesting E-Tugra's understanding was correct, but rather, because I'm wanting to better understand how E-Tugra reached a wrong understanding, both to ensure that there are mitigations in place for the correct understanding, and because it suggests there may be other misunderstanding about expectations.

This is a hugely critical, security relevant expectation of CAs, that has been discussed for years, and so while E-Tugra is confident that nothing untoward happened, the concern here is about how all of that was missed.

Equally, the suggestion of pre-BR mistakes and misunderstandings seems to highlight the critical importance of replacing the existing E-Tugra root with a new hierarchy, so that these past mistakes and misunderstandings do not continue to expose users to risk. Some discussion was had on dev.security.policy, some recently at the CA/Browser Forum F2F, and it seems like there is an opportunity here for E-Tugra to more comprehensively make meaningful steps here, if they're going to continue to want to participate as a TLS CA.

In a Chrome capacity, I have not seen any discussion yet about E-Tugra's plans for revoking these certificates, which is the expectation; OneCRL does not protect Google users, for example, and it seems E-Tugra needs to discuss with all root programs their lack of BR compliance, because the risk applies to all entities that include E-Tugra. The BRs state revocation is expected, so if E-Tugra is not revoking, that requires its own discussion.

Flags: needinfo?(dtokgoz)

Thanks Ryan for the feedback. I would like to give some history before we get into more details on this issue. This CA was issued in 2013 and BR requirement was put into action in 2016. When this requirement was mandated we worked immediately internally on our certificate issuance processes and ensured that there is no way to issue any TLS certificate from this CA because this CA was used to issue end entity qualified signature certificates meeting the ETSI requirements. We ensured that our processes are strong enough and it would not let any TLS certificate issued from this CA. Even CT Logs can ensure that any TLS certificate was not issued from this CA.
Moving forward, we created new intermediate CAs after 2018 and we applied for inclusion of them in a week after issuing as expected and include them on all audit reports after as expected in policies and BR.
We are doing an audit of this CA every year according to ETSI auditing requirements but TLS was out of scope of the audit as this aforementioned CA was not meant to issue any TLS certificate based on the actions taken in 2016. In all policy and management reviews, for this CA, it was always assumed that it was out of scope, because it was created before and also audited by another authority. This aforementioned CA was not available in CCADB records for several years.
We didn’t announce it to anyone because we assume that BR requirement was applied in 2016 and this CA was issued before that and it can’t cause any risk because this CA didn’t issue any TLS certificate. Even when we issued a new chain, we didn’t issue any certificate from this CA. We accept that this CA audit was not done focusing on TLS context so we have decided to do an independent audit from 2016 to strengthen our applied measurements that have no TLS certificate issued from this CA.
Based on discussions in different threads suggesting the solutions to avoid the risk for this issue we have already taken steps and applied for Mozilla OneCRL and working with other root programs i.e. CRLSet so no one would be affected from this CA due to BR non-compliance requirement.
We are doing our audit at the start of August and we’ll ensure that CA is added in the Audit report also.
e-Tugra aim is to be fully compliant with BR, ETSI and other standardized procedures and processes to avoid any such issues in the future. We revise our processes and do measures based on the changes in the standards to be compliant with them.
We will comment with an updated Incident Report in a few days.

Flags: needinfo?(dtokgoz)

Thanks.

To make sure there's no misunderstanding: Chrome does not, and has never, accepted retroactive audits. CA certificates that are technically capable (according to the BRs definition) of causing TLS issuance, and that have not been audited, are unconstrained intermediates and expected to be revoked. While we understand there may be delays to revocation, it's unacceptable for such delays to be indefinite: in those cases, we look to remove the root CA and any certificates they operate.

You can and should feel free to direct any Chrome-specific questions to chrome-root-authority-program@google.com , but this CA has been operated outside of the ongoing requirements of the BRs. As part of your incident report promised in Comment #9, you should have a clear plan: to either replacing this (and all other) unaudited intermediates, or to rotating your root certificate, so that we can remove trust in your existing legacy root. For example, transitioning all your existing TLS customers to your new hierarchy "immediately" (i.e. existing customers, not just new customers), so that the TLS capability can be removed from your existing root without negatively impacting your qualified certificates, which are presumably the more difficult of the two to replace.

Flags: needinfo?(dtokgoz)

We are definitely aware about chrome and all other root programs requirements. From the day one, when this bug was created our teams is working hard in finding the best solution to address this point.

In addition, adding this CA to OneCRL and including it in audit reports, we are trying to create the best solution. We even working and in replacing all certificates for this aforementioned CA. Such solutions require approval process from related Turkish authority and absolute confident that it will work as expected. We are also communicating with respective authorities and 3rd parties that trust us for Qualified Signature Certificates. To make it smooth execution by getting confidence of respective authorities and third parties it’s taking some more time for finalizing the incident report.

One of the options that we work on and you mentioned in your comment is moving SSL certificates to another hierarchy. We created new roots on march 2020 and they were audited and are in the inclusion process. When it will be included, moving all TLS certificates seems one of the easiest solutions for us.

Flags: needinfo?(dtokgoz)

To keep this thread updated about the progress, we are making for this point. We are communicating and working with desired third parties, authorities. But there are one-week nation wide holiday. So it’s really difficult to get some feedback from them. We continue with very limited communication. We would be posting further progress update.

Flags: needinfo?(dtokgoz)

Hi,
An update for the incident is as follows.
** The inclusion of Aforementioned SubCA in OneCRL was completed with the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1719683
** Last week, E-Tugra was in audit and auditors reviewed our hierarchies. With this new audit activity, the auditors will include this sub-CA in the audit letter that will indicate and ensure that there are no TLS certificates issued from this sub-CA.
** In parallel, we are doing hard work on revoking this aforementioned SubCA with the scenarios explained below:

  • Cross Signing the aforementioned SubCA with a new root for managing existing Qualified Signature Certificates. We are working on it with many 3rd parties, about the possibility of senario and updates of thousands softwares on production
  • Replacing all end entity certificates which are in smartcards of this Aforementioned SubCA
    Our aim is to clarify the deadlines of the tasks and revocation process, by the end of this month hopefully. And complete the revocation before the beginning of 2022.
Flags: needinfo?(dtokgoz)

An update for the incident is as follows. In the previous update, we indicated that the inclusion of Aforementioned SubCA in OneCRL was completed..E-Tugra was in audit and auditors reviewed our hierarchies with inclusion of this sub-CA in the audit.
We are continuing with the scenarios explained below to be revoking this aforementioned SubCA .

  • Cross Signing the aforementioned SubCA with a new root for managing existing Qualified Signature Certificates.
  • Replacing all around a hundred thousands end entity certificates which are in smartcards of this Aforementioned SubCA. We are buılding a team that will be responsible for this migration. They started to reach customers and replace the certificates.
    We planned to complete the revocation at the end of this December.
    Ben: Is it possible to set the next update date of this bug as the end of December?
Flags: needinfo?(bwilson)

Let's set the next update for 01-Nov-2021 so that we can get a status update.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] → [ca-compliance] Next update 2021-11-01

Bug Update: As we indicated previously we follow 2 independent action to revoke subCA.

  • When testing "Cross Signing the aforementioned SubCA with a new root for managing existing Qualified Signature Certificates", we faced some problem on some application libraries used by 3rd parties. We will continue to work.
  • For action "Replacing all around a hundred thousands end entity certificates which are in smartcards of this Aforementioned SubCA.", We completed the migration of over %35 of Qualified Certificates issued by this CA until now. The biggest problem is to trigger the certificate owners to start migration action on their smartcards. We continue the process as planned.
    We hope to complete the actions at the end of December. Our aim is to complete migration of certificates as much as possible.
    Ben: Is it possible to set the next update date of this bug as the end of December?
Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] Next update 2021-11-01 → [ca-compliance] Next update 2022-01-03

Pre-Final Report for revocation of SubCA.
As known, we planned to migrate all certificates issued by this CA to the new root created for Qualified Certificates with ECC. About %60 of migration was completed until today. Most of the remaining certificates either are not used or we cannot reach the owners via any channels. There are about a small set of certificates remaining that must be migrated. The owners of these certificates are Nationwide Banks and they could not complete the migration of their infrastructures that have the capability usage of ECC algorithms. They need an additional 15 days to complete their migrations. After completing these migrations by them, we will revoke the CA on January 15th - 20th.
We believe that this minor change in the plan will be acceptable and will be met with understanding.

Update: We are following the plan that is revocation of the CA before January, 20th

Whiteboard: [ca-compliance] Next update 2022-01-03 → [ca-compliance] Next update 2022-01-20

The aforementioned intermediate CA with serial number "1f0ec403b3801cad" was revoked on Jan 19th, 2022 and the root crl was published on http://crl.e-tugra.com/etugra_root.crl .

I will close this bug on next Wed. 6-Feb-2022 unless there are issues or items still to discuss.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] Next update 2022-01-20 → [ca-compliance] Next update 2022-02-16
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next update 2022-02-16 → [ca-compliance] [audit-failure]
You need to log in before you can comment on or make changes to this bug.