1801345 - E-Tugra: Incident Report (Security Issues)

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ahmed]

Attachment: Incident-Report.pdf

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0

Comment 1

[Ahmed]

1 . How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.
An email from Ian Carroll <[e-mail]> on Nov 13, 2022.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Nov 13, 2022: An email from Ian Carroll about finding
Nov 13, 2022: The issues were fixed immediately
Nov 14, 2022: We sent an email to the reporter for our progress.
Nov 14, 2022: Began to analyze both the impacts and reasons. All user activities are traced from the last one year.
Nov 17, 2022: Reporter published a blog about this issue.
Nov 18, 2022: Incident Report is published with this text with Mozilla Incident Report Format.
Our security team started work immediately on the reported issues along with a full scan of all the systems.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
No certificates are involved in the incident. CA continued to serve as expected.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
No certificates are involved in the incident. CA continued to serve as expected.
Customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition.
Only three documents related to some users' (non SSL) agreements are accessed by the reporter.
5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
No certificates are involved in the incident.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The information shared by the reporter is for an internal application. Restricted access policy was not applied well on the application so the issue is based on violation of access policy. Issue was fixed immediately and on a safe side complete system scanning activities were performed to ensure no other applications were affected. No sensitive information disclosure revealed in this issue.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
We have attempted remediation actions to overcome this issue as part of full system scanning. We have a process in place for the penetration and vulnerability assessment of the system which was executed on the planned time. In addition to another cycle of penetration/vulnerability exercise we are closing gaps in the access policy management process and audits.
We aim to execute these improvements by the end of this year.

Comment 2

[Ahmed]

Information about the incident is available at:
https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s

Comment 3

[Chris Clements]

Thank you to Ian Carroll for identifying these concerns.

Ahmed, we have a few initial questions to help us better understand the scope and perceived impact of what is being reported. Additional questions will likely surface as more information becomes available.

1. The timeline of actions in #2 must be updated to include timestamps, and should account for relevant events before the initial communication from the security researcher such as:

   • when each issue was introduced into the environment (answering the question “for how long as this vulnerability existed?”)
   • when the last penetration test, risk assessment, and qualified audit of the publicly-trusted hierarchy were each performed.

2. The phrase “The information shared by the reporter is for an internal application.” could suggest that the system the security researcher accessed is completely separated from E-Tugra’s publicly-trusted hierarchies, i.e., the certificate hierarchies terminating at the following root CAs:

   CN=E-Tugra Certification Authority,OU=E-Tugra Sertifikasyon Merkezi,O=E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş.,L=Ankara,C=TR
   Certificate Hash: B0BFD52BB0D7D9BD92BF5D4DC13DA255C02C542F378365EA893911F55E55F23C
   SPKI Hash: c1ad1b1898ec395048df070bfa217e25c913bed8ca6b73de085528846a0103c1

   CN=E-Tugra Global Root CA ECC v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
   Certificate Hash: 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13
   SPKI Hash: 56f2ea8684106d0c73333951059d2bff9ace0a9934f015c5d84c5a959dfbb3cc

   CN=E-Tugra Global Root CA RSA v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
   Certificate Hash: EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2
   SPKI Hash: b3effbf46bcf66aedf71427e6bd60bf1a1878c7b72cab178703485fda6e3db38

   • Please confirm this interpretation is correct (i.e., what the researcher accessed is separate from the systems related to the hierarchies above).

3. If the above interpretation is correct, describe, in detail, how the hierarchies stated above are separated from the applications and corresponding systems accessed by the security researcher.

   • Minimally describe whether the compromised systems and those related to the publicly-trusted hierarchies above are:
     • physically separated?
     • logically separated?
     • “air-gapped'' from one another, or otherwise segmented such that there’s no possible route between the two?

4. Describe, in detail, the security controls in place that would have prevented an adversary from traversing systems/networks to gain access to systems or applications that correspond to E-Tugra’s publicly-trusted CA hierarchies. Provide sufficient detail such that we can objectively evaluate them as being satisfactory safeguards against the vulnerabilities described by the security researcher.

5. Describe the maximum level of access or functional privilege an unauthorized third party could have exercised through the attack, especially concerning E-Tugra’s publicly-trusted CA systems or related PKI components (e.g., could an adversary use their access to request new certificates, intercept domain validation random values, perform certificate management functions, remotely manage network-attached HSMs, exfiltrate subscriber data, etc.?)

6. Describe whether the vulnerabilities detailed by the security researcher also existed on E-Tugra’s publicly-trusted CA systems or related PKI components. What evidence does E-Tugra plan to present to their auditor in response to this incident to corroborate this statement?

7. Describe the steps taken to evaluate whether the publicly-trusted hierarchies and corresponding systems capable of impacting certificate issuance and management are unaffected by the vulnerabilities described by the security researcher.

8. How did you verify “customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition” for all publicly trusted TLS certificates issued by E-Tugra?

9. In the remediation steps you stated another cycle of penetration/vulnerability exercises.

   • When will these exercises occur?
   • Who will perform the exercises and will they attest to the completeness of scope including all system elements that interact with E-Tugra’s publicly-trusted hierarchies?
   • What are the assessor’s qualifications?
   • What is the assessor’s independence?
   • How will the resulting report(s) be shared?

Comment 4

[Ahmed]

Hi Chris Clements,

We are working with several internal teams to get their feedback and analysis on the problems being reported by researcher.

We are compiling a detailed report based on feedback from each team before we publish it. We hope that we will able to complete this activity within a week time.

Comment 5

[Ahmed]

Hi Chris,

Thanks for the queries. Before we give the detailed feedback on the questions, it is better to share some insight which will be surely helpful to digest the information against answers below. Both non-SSL applications and CA systems are physically and logically separated systems. We have dedicated teams for managing them. In order to access the CA system we have a proper dedicated physical M of N rule with multi-factor authentication applied. Same approach is applied on HSM access also. Even HSMs and CAs are also isolated. We do an annual CA external audit and our auditing company is a European company. Turkish audit company not selected intentionally to be biased or influenced. Every year a full physical site audit is performed by the auditor for the CA system. In our recent audit, auditors visited our site physically and they observed all the explained operations by full auditing CA system. Our CA logging has a signed timestamped and non-tamper system. Auditors reviewed our CA logging system and ensured it was non-tampered. This incident happened on another system as one of our internal application developed (tracking reseller pre-application activities) published a month ago about non-SSL business.

Please see inline-responses below against each query:

1. The timeline of actions in #2 must be updated to include timestamps, and should account for relevant events before the initial communication from the security researcher such as:
   o when each issue was introduced into the environment (answering the question “for how long as this vulnerability existed?”)
   [e-Tugra]: Details of actions taken:
   ▪ Nov 13, 2022 15:11 (UTC +3): An email from Ian Carroll about finding
   ▪ Nov 13, 2022 15:26 (UTC +3): All teams in e-Tugra were notified and started work on the issue immediately
   ▪ Nov 13, 2022 15:44 (UTC +3): The issues were fixed immediately, by closing related sites
   ▪ Nov 14, 2022 09:50 (UTC +3): Additional email came from the reporter
   ▪ Nov 13-14, 2022: Our all team started to scan all systems and completed. The affected applications were closed.
   ▪ Nov 14, 2022 19:36 (UTC +3): We sent an email to the reporter for our progress.
   ▪ Nov 15, 2022:
   ● e-Tugra teams started to analyze both the impacts and reasons. All user activities were tracked
   ● We have executed a comprehensive exercise for all our services (both public and internal)
   ● All access rules and implementation are reviewed
   ● All applications and services are checked with cross control for the authentication and authorization rule
   ● Checked the CA system and didn’t find any impact for mis-issuance / reissuance of certificate, logging component etc.
   ▪ Nov 17, 2022 13:27 (UTC +3): Reporter sends notification about his blog.
   ▪ Nov 17, 2022 13:27 (UTC +3): Reporter published a blog about this issue.
   ▪ Nov 18, 2022 19:55 (UTC +3): Incident Report was published on Bugzilla.
   Our security team continued work on:
   ▪ Improvement on security control to prevent such issues
   ▪ Internal security test and scan, cross checks all access rules
   ▪ Starting third party penetration test action with full coverage
   ▪ Revising procedures and fix the gaps if any
   o when the last penetration test, risk assessment, and qualified audit of the publicly-trusted hierarchy were each performed.
   [e-Tugra]: CA system penetration testing was performed last year based on the recommendations from CA/B network security guidelines. Penetration testing report was presented to auditors like yearly audit. Risk assessment is also practiced annually or whenever there is a change that requires “risk review” in the system/procedures. We performed the planned activity this year and it was approved by the e-Tugra security board. We have gone through the ETSI TSP Audit in August 2022.

2. The phrase “The information shared by the reporter is for an internal application.” could suggest that the system the security researcher accessed is completely separated from E-Tugra’s publicly-trusted hierarchies, i.e., the certificate hierarchies terminating at the following root CAs:
   CN=E-Tugra Certification Authority,OU=E-Tugra Sertifikasyon Merkezi,O=E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş.,L=Ankara,C=TR
   Certificate Hash: B0BFD52BB0D7D9BD92BF5D4DC13DA255C02C542F378365EA893911F55E55F23C
   SPKI Hash: c1ad1b1898ec395048df070bfa217e25c913bed8ca6b73de085528846a0103c1
   CN=E-Tugra Global Root CA ECC v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
   Certificate Hash: 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13
   SPKI Hash: 56f2ea8684106d0c73333951059d2bff9ace0a9934f015c5d84c5a959dfbb3cc
   CN=E-Tugra Global Root CA RSA v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
   Certificate Hash: EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2
   SPKI Hash: b3effbf46bcf66aedf71427e6bd60bf1a1878c7b72cab178703485fda6e3db38
   [e-tugra]: Already explained above, the CA system is both physically and logically separated.
   o Please confirm this interpretation is correct (i.e., what the researcher accessed is separate from the systems related to the hierarchies above).
   [e-Tugra]: We confirm that the affected system is separate from the CA system.

3. If the above interpretation is correct, describe, in detail, how the hierarchies stated above are separated from the applications and corresponding systems accessed by the security researcher.
   o Minimally describe whether the compromised systems and those related to the publicly-trusted hierarchies above are:
   ▪ physically separated?
   ▪ logically separated?
   ▪ “air-gapped'' from one another, or otherwise segmented such that there’s no possible route between the two?
   [e-Tugra]: CA systems have separated physical locations that have high zone security physical and logical access controls. They have separate firewalls that apply very high restrictions and own log systems.

4. Describe, in detail, the security controls in place that would have prevented an adversary from traversing systems/networks to gain access to systems or applications that correspond to E-Tugra’s publicly-trusted CA hierarchies. Provide sufficient detail such that we can objectively evaluate them as being satisfactory safeguards against the vulnerabilities described by the security researcher.
   [e-Tugra]: CA system is not exposed publicly and it has a separate network managed in a privately owned physical data center and even e-Tugra CA staff can not access any or traverse any machine in the CA system without physical M of N rule and multi-factor authentication principle in place. We have proper audit logs as well who entered into CA systems and logs are non-tampered.

5. Describe the maximum level of access or functional privilege an unauthorized third party could have exercised through the attack, especially concerning E-Tugra’s publicly-trusted CA systems or related PKI components (e.g., could an adversary use their access to request new certificates, intercept domain validation random values, perform certificate management functions, remotely manage network-attached HSMs, exfiltrate subscriber data, etc.?)
   [e-Tugra]: As mentioned above, CA systems can not be accessed by unauthorized third parties. Remote access to network attached HSMs in the CA system is not possible because they are isolated. Security researcher attempted to get domain validation certificates multiple times but could not get it because of secure access control applied and he confirmed it in the blog. For domain validation values can only be accessed by trusted CA staff after authentication rules explained above. CAs, HSMs and related logs can’t be accessed remotely as it requires physical presence based on defined procedures above.

6. Describe whether the vulnerabilities detailed by the security researcher also existed on E-Tugra’s publicly-trusted CA systems or related PKI components. What evidence does E-Tugra plan to present to their auditor in response to this incident to corroborate this statement?
   [e-Tugra]: Like every year, we will present all the sites, applications, network logs, system topologies, internal security controls and pen testing reports etc. to the auditor. Even auditors can ask anything related during auditing activities.

7. Describe the steps taken to evaluate whether the publicly-trusted hierarchies and corresponding systems capable of impacting certificate issuance and management are unaffected by the vulnerabilities described by the security researcher.
   [e-Tugra]: We checked the CA system and all the non-tampered timestamped logs to find any attempt for the mis-issuance of the certificate or unauthorized access in the CA system.

8. How did you verify “customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition” for all publicly trusted TLS certificates issued by E-Tugra?
   [e-Tugra]: Strong encryption is being applied to protect the customer's sensitive information for TLS certificates and no sensitive information is logged in the logging system.

9. In the remediation steps you stated another cycle of penetration/vulnerability exercises.
   o When will these exercises occur?
   [e-Tugra]: We have already initiated the process with a third party penetration testing company. Once the plan is finalized we’ll share it here. We aim that it will be started as soon as possible.
   o Who will perform the exercises and will they attest to the completeness of scope including all system elements that interact with E-Tugra’s publicly-trusted hierarchies?
   [e-Tugra]: We already shared the complete requirements which covers the internal, external infrastructure, DDOS attacks, web application etc. penetration testing. Each area has its own rules & practices which will be performed as part of this exercise by third party pen testers.
   o What are the assessor’s qualifications?
   [e-Tugra]: Third party pen testers has got following certifications /skillset:
   ● Certified Information Systems Security Professional (CISSP)
   ● Offensive Security Certified Professional (OSCP)
   ● Certified Secure Software Lifecycle Professional (CSSLP)
   ● Certified Ethical Hacker (CEH)
   ● ElearnSecurity Web application Penetration Tester (eWPT)
   ● Practical Network Penetration Tester (PNPT)
   ● Offensive Security Wireless Professional (OSWP)
   ● Certified Secure Application Programmer (C-SAP)
   ● Microsoft Certified Solutions Associate (MSCA)
   ● Linux Professional Institute Certification (LPIC)
   o What is the assessor’s independence?
   [e-Tugra]: Penetration testing company is an independent third party. They will be executing the complete exercise.
   o How will the resulting report(s) be shared?
   [e-Tugra]: They will be sharing the penetration testing reports once they are done with the exercise. We’ll present reports to the auditors like we presented earlier.

Comment 6

[Ben Wilson]

I'd like to add some follow-up questions to those from Chris Clements in Comment #3 :

10. Regarding the timeline of events before the initial communication from the security researcher:

a. When were the prior vulnerability scans and penetration tests performed?

b. Did the scope of prior vulnerability scans and penetration tests include non-CA systems, given that advanced persistent threats often leverage office systems and use them to hop into CA systems? If no, then why not, and if yes, then which systems?

c. When was the last risk assessment performed and comprehensive security plan updated, both of which are required annually by Section 5 of the CA/Browser Forum’s Baseline Requirements?

11. Regarding the scope of e-Tuğra's upcoming vulnerability scans and penetration tests:

a. How will we know whether testing was adequately scoped to include systems that interact with e-Tuğra’s certificate-issuing systems, e.g. RA systems, customer portals, and other related non-CA components?

b. Will testing be white box, black box, gray box, or a combination of the three types?

c. What is the reference source or catalog of vulnerabilities that testing will examine?

12. Regarding the results from e-Tuğra's vulnerability scans and penetration tests:

a. How will we know that detected vulnerabilities have been remediated or that a remediation plan has been adopted and is being implemented? (We assume that you will be documenting the rationale for determining that a vulnerability did not require remediation, as required by section 4.f.3 of the Network and Certificate System Security Requirements.)

b. After remediating the vulnerabilities detected by the upcoming testing exercises, will you re-test for vulnerabilities and confirm that they have all been remediated?

Comment 7

[Ahmed]

Hi Ben,
Thanks for the comment. Please see [e-Tugra] feedback against each point below:

10. Regarding the timeline of events before the initial communication from the security researcher:
    a. When were the prior vulnerability scans and penetration tests performed?
    [e-Tugra]: We do vulnerability scans monthly and penetration tests annually. We present the reports to auditors in the annual audit.
    We executed the vulnerability scan in October 2022 and Penetration Test in September 2022, before the current incident was reported. We have scheduled another cycle of penetration testing in the coming months. We will share the reports with auditors.

b. Did the scope of prior vulnerability scans and penetration tests include non-CA systems, given that advanced persistent threats often leverage office systems and use them to hop into CA systems? If no, then why not, and if yes, then which systems?
[e-Tugra]: Both CA and non-CA systems were in the scope of previous vulnerability and penetration exercise. The exploited application was published publicly a month ago prior to incident

c. When was the last risk assessment performed and comprehensive security plan updated, both of which are required annually by Section 5 of the CA/Browser Forum’s Baseline Requirements?
[e-Tugra]:We follow the CA/B baseline requirements section 5 for the annual risk assessment. Additionally we do whenever there is some change which requires risk assessment to be performed. We recently performed a risk assessment activity in February 2022.

11. Regarding the scope of e-Tuğra's upcoming vulnerability scans and penetration tests:
    a. How will we know whether testing was adequately scoped to include systems that interact with e-Tuğra’s certificate-issuing systems, e.g. RA systems, customer portals, and other related non-CA components?
    [e-Tugra]: We can send the scope to the browser's program main contact personnel and will present the scope and reports to auditors in our next audit. We thought that sharing scope details publicly is not a good option.
    In this penetration and next all systems that we have CA and non-CA will be covered.

b. Will testing be white box, black box, gray box, or a combination of the three types?
[e-Tugra]: Yes, all three types will be covered in the upcoming testing scope.

c. What is the reference source or catalog of vulnerabilities that testing will examine?
[e-Tugra]: Penetration testing company has already the list of sources and catalog as part of their testing processes. We can share the catalog penetration testing company will use.

12. Regarding the results from e-Tuğra's vulnerability scans and penetration tests:
    a. How will we know that detected vulnerabilities have been remediated or that a remediation plan has been adopted and is being implemented? (We assume that you will be documenting the rationale for determining that a vulnerability did not require remediation, as required by section 4.f.3 of the Network and Certificate System Security Requirements.)
    [e-Tugra]: Like previous vulnerability scans and penetration testing exercises, we will create a report against each vulnerability and present it to the auditor. These reports cover how and when they are fixed. If a remediation is not desired, the reasons and risks are reported. We are following the recommendations mentioned in the Vulnerability Detection and Patch Management section of Network and Certificate System Security Requirements.

b. After remediating the vulnerabilities detected by the upcoming testing exercises, will you re-test for vulnerabilities and confirm that they have all been remediated?
[e-Tugra]: Yes, the testing scope covers the re-validation of fixed remediation.

Comment 8

[Ben Wilson]

I'm setting the next update on this bug to January 6, 2023, with the expectation that e-Tuğra can provide an update on its penetration and vulnerability testing.

Comment 9

[Ryan Dickson]

Hi Ahmed,

While reviewing the images on Ian’s blog, I noticed what appeared to be dnsNames in this screenshot.

For example, in the “Title” column, we see the following dnsNames appear in what seem to be messages to subscribers confirming certificate issuance (i.e., the title contains the phrase "Your SSL Certificate Is Ready"):

• bakbuonemli.com,
• *.kolaykvkk.com.tr,
• ankarabozkurtnakliyat.com,
• esperdetasarim.com, and
• tamdosya.com.tr

In all of the above cases, the dnsNames correspond with a pre-certificate issued by either:

• CN = TrustSafe TLS RSA SubCA R1, O = Isimtescil Bilisim A.S., C = TR, or
• CN = E-Tugra TLS RSA SubCA R1, O = E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI, C = TR

Both of these issuing CAs chain to CN = E-Tugra TLS RSA CA R1, O = E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI, C = TR (1 and 2), and eventually CN=SSL.com Root Certification Authority RSA, O=SSL Corporation, L=Houston, ST=Texas, C=US, meaning these certificates are publicly-trusted.

Studying the timing of the entries in the image described above against the validity dates included in the pre-certificates listed above presents a pretty consistent delta between the time of pre-certificate issuance and customer notification (about 11 minutes).

Domain	Crt.sh issuance time (GMT)	Ian’s Blog Image Time (GMT +3)	Ian’s Blog Image Time (Adjusted to GMT [-3])	Difference
bakbuonemli.com	11/13/2022 11:10:18	11/13/2022 14:21:32	11/13/2022 11:21:32	11 minutes and 14 seconds
*.kolaykvkk.com.tr	11/13/2022 08:22:46	11/13/2022 11:33:56	11/13/2022 08:33:56	11 minutes and 10 seconds
ankarabozkurtnakliyat.com	11/13/2022 08:00:11	11/13/2022 11:11:25	11/13/2022 08:11:25	11 minutes and 14 seconds
esperdetasarim.com	11/13/2022 07:11:43	11/13/2022 10:22:54	11/13/2022 07:22:54	11 minutes and 11 seconds

This timing consistency seems to suggest some degree of connectivity (e.g., an automated push) between the issuing CA and the system Ian accessed, which I interpret to mean the system described in Ian's blog post is not “separate” from the CA system (contradicts the following points made in Comment 5:

[e-tugra]: Already explained above, the CA system is both physically and logically separated.
and
[e-Tugra]: We confirm that the affected system is separate from the CA system.

A few questions:

1. Is the above understanding correct? (i.e., there is some connection between the system Ian accessed and the issuing CA represented in the pre-certificates above that allows it to become aware of certificate issuance)
2. Can we expect a discussion and corresponding evaluation of this connection in the penetration test?
3. Comment 7 states:

[e-Tugra]: We do vulnerability scans monthly and penetration tests annually. We present the reports to auditors in the annual audit.
and
The exploited application was published publicly a month ago prior to incident.

However, it's not clear if "published" means "first deployed", "first made internet accessible", or something else. Can you clarify your intended use of "published"?

Thanks,
Ryan

Comment 10

[Ahmed]

Hi Ryan,

Thanks for your comment. Please see [e-Tugra] feedback against each point below:

A few questions:
Is the above understanding correct? (i.e., there is some connection between the system Ian accessed and the issuing CA represented in the pre-certificates above that allows it to become aware of certificate issuance)
[e-Tugra]: We stated earlier that the accessed application was intended for internal reporting usage. Once the CA system has performed its operations it then passes the post certificates specific log information using one-way flow secure network connection to the reporting system. The application classifies it based on defined rules to generate reports. It doesn’t have any direct operational connection with the CA system for the certificate life cycle process as domain validation emails are sent by the separated CA system and there is no automated way for the certificate issuance.

Can we expect a discussion and corresponding evaluation of this connection in the penetration test?
[e-Tugra]: We have enhanced the scope to both CA and non-CA systems in the penetration testing exercise. Penetration testing firm will be evaluating both systems and identifying any gaps if any.

However, it's not clear if "published" means "first deployed", "first made internet accessible", or something else. Can you clarify your intended use of "published"?
[e-Tugra]: We apologize for the inconvenience caused due to word selection. This application made the internet accessible for the first time mistakenly.

Comment 11

[ian]

We stated earlier that the accessed application was intended for internal reporting usage. Once the CA system has performed its operations it then passes the post certificates specific log information using one-way flow secure network connection to the reporting system.

I am a little worried that we've gone from no connection to "one-way flow" connection, and I do not think that is true either. The emails in the administrative system that was accessed were related to the e-Tugra customer portal. The customer portal is intrinsically used to purchase certificates by customers, so orders must somehow flow from the customer portal to the issuance infrastructure, and then back again once issued. I do not see how it is possible for there to only be a "one-way flow".

As I stated on m.d.s.p., it is a common feature among CAs that customer portals can re-use existing validations, or re-issue existing certificates, without completing domain-control validation again. It would be good to know if e-Tugra's customer portal could do this, as it would have allowed an attacker to re-issue any existing e-Tugra certificate if so.

Comment 12

[Ahmed]

Hi Ian,

The reporting system receives the information from the CA system via some secure tunneling and generates emails/notifications not only for certificate related processes but also for other applications (resellers, internal applications etc.). This application does not participate or play any role in the certificate issuance process instead emails or reporting.

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates but the accessed application does not have any impact in this process.

Comment 13

[ian]

Hi Ahmed,

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates but the accessed application does not have any impact in this process.

The e-Tugra customer portal allows resetting account passwords via email. The password reset codes were then displayed in the accessed application. As you can see in my post, there are emails of "Password Reset Request" visible in the application.

As a result, anyone who accessed the affected application could have logged into any e-Tugra account, and (it appears) then re-issued any e-Tugra certificate.

Comment 14

[Ahmed]

Hi Ian,

All DV validation and issuing certificate life cycles are processed by a separate system after request validation so this system does not hold any impact for the certificate life cycle.
We checked in our logging system and you attempted multiple times to place an order for the certificate through the portal but could not get the certificate. Now this shows that you want to get more CA system details to emphasize your point of interrupting the certificate life cycle process but it could not happen even with your attempts. We clearly state that there is no connection of the accessed application to the CA system for the certificate life cycle process.

Once this incident was reported we immediately reacted and reported on bugzilla (with detailed incident report) to inform related parties and even informed our customers about this incident and no DV certificate being re-issued as part of this incident. We thank you for reporting this incident.

We mentioned above that we are doing a comprehensive penetration testing and we will be sharing the penetration testing results with the auditors as we do it each year.

Comment 15

[ian]

Hi Ahmed,

I am not sure what your comment is trying to refute. You previously said:

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates

The vulnerabilities I discovered allowed me to view the password reset emails sent to any customer. As a result, I could log into any customer account, via resetting their password. There is evidence for this in the screenshots of my post. Based on what you have said, this could have compromised the certificate of any E-Tugra customer.

The penetration test and incident report are fine. I am only concerned that E-Tugra does not understand the impact of a severe issue in its own systems. It is important that we are clear that this incident could have impacted E-Tugra's issuance and CA operations. The following statements from E-Tugra appear to be simply false:

We confirm that the affected system is separate from the CA system.
It doesn’t have any direct operational connection with the CA system

Based on the above, it is impossible for the affected system to be completely separated from the CA system. The customer portal cannot be separate from the CA system but also the main way for users to request and obtain certificates.

We checked in our logging system and you attempted multiple times to place an order for the certificate through the portal but could not get the certificate.

I was unable to purchase a certificate in my own account because your website did not accept American credit cards. This is not relevant to the vulnerabilities.

Comment 16

[Ahmed]

Hi Ian,

We understand the concerns raised in the incident report and potential impact on certificate issuance and CA operations. We performed several steps to address the issue, including remediation measures and making improvements to our systems / processes to ensure the security and integrity of our operations. We executed a thorough investigation of the incident and have found no evidence that the vulnerability was exploited by any malicious actors. We checked all the logs and each of our issued certificates to find any negative effect on our system. We have taken preventive actions and supportive measures to avoid such vulnerabilities in future.

We understand the importance of maintaining the security and integrity of our customers' certificates, and we assure that we are taking care of best security principles in place to ensure the protection of our customers' data and assets. Thanks for your critical input as such feedback is always helpful in strengthening security further.

Comment 17

[Ben Wilson]

Hi Ahmed,
Where are we on the remediation measures related to this matter? What action items are still being worked on?
Thanks,
Ben

Comment 18

[Ahmed]

Hi Ben,

We have taken the following remediation measures as part of this incident:

Monitoring processes were enhanced in operations
Internal Tests are enhanced and performed regularly for both CA & Non-CA systems.
Organized Security Training sessions for CA staff
Revised applications acceptance procedure and tests

The only action item which is in-progress is Penetration Testing. We were in discussion with several third party penetration testing companies. We have selected the better one and shared the comprehensive scope with them. Penetration testing company has already started the work. We will keep posting about the updated progress for it.

Comment 19

[Kathleen Wilson]

Ahmed, Please add status updates every week from now until this is fully resolved.

Comment 20

[Ahmed]

Hi Kathleen,

Apology for not posting updates frequently as we were aggressively working with penetration company considering the importance of this incident. Of-course I will be posting updates as you suggested.

Penetration company has executed the major part of the scope with some minor items in-progress. We got some minor findings reported, fixed them and shared the remediation plan with them. We are expecting they will be closing the scope this week and proceed with the remediation/validation of the fixed points.

In the reports there are no findings surrounding this incident and we are hopeful that results will be very much positive here after the completion of this exercise.

We will share the findings and related reports to auditors during the audit. It would be great if you guide us which points we can share here for the community & google group discussion to close this incident.

Comment 21

[Ahmed]

Hi,

I am writing to inform you of the results of the penetration testing we conducted on our systems and provide an update on the actions we have taken in response. We applied several security controls in strengthening both internet and intranet systems and penetration testing were performed after we were done with our updates.

Penetration testing company is almost done with the exercise and a couple of areas are left for the intranet tasks. They are also starting the remediation testing for the provided fixes by the end of next week.

We aim that the whole exercise will be completed in this month. We will keep posted updates. Thanks.

Comment 22

[Ben Wilson]

Hi Ahmed,
Please make sure that there is some type of executive summary or publicly available report generated by the pentesting company that provides a high-level, non-security-sensitive explanation of what has been accomplished. We need something that can be uploaded here as an attachment.
Thanks,
Ben

Comment 23

[Ahmed]

Hi,
An update for the pen test process:

• First phase (covering the full scope) has been completed by pen test company
• Fixes for some minor findings are also delivered to pen test company
• Pen test company will start the validation phase next week
  The pen testing company will prepare an executive summary report independently for publishing to the community once they are done with the validation phase.
  Thanks,
  Ahmed

Comment 24

[ryan_hurst]

I noticed that according to the timeline, the penetration test started 131 days ago (Nov 18, 2022). A typical penetration test usually lasts a matter of weeks, not nearly half a year. It is difficult to comprehend what a penetration testing firm would be doing for such an extended duration, and I sincerely hope the executive summary of the engagement carries enough detail to clarify this.

One plausible explanation for the long timeline could be an interim remediation and retesting process. If this is the case, it would be valuable for stakeholders to understand the type of fortification that was performed during this period. Additionally, this raises the question of why the annual penetration testing missed these issues in the first place. It would be beneficial for the community if an explanation could be provided on this front, along with a discussion of the changes being implemented to prevent such lapses from happening again in the future.

As we all share a common interest in maintaining a secure and robust ecosystem, it is crucial to address these concerns and ensure transparency in the process. I look forward to hearing more about the specifics of the engagement, the reasons for the extended timeline, and the steps being taken to enhance the overall security posture of the organization.

Comment 25

[Ahmed]

We fully understand the importance of this topic and appreciate the efforts made to keep everyone updated on the progress.

It is commendable that this penetration testing activity was not just a routine annual test activity rather a need-based. We were in discussions with a third-party penetration testing company. It is customary when working with third party companies, we had to align our timelines according to their schedule. We explained earlier that we expanded the scope of the testing, which required further coordination with the testing team.

We are pleased to report that the pen testing team worked diligently, revising their testing scripts and suggesting additional points to ensure comprehensive testing. They were also available for execution, and work commenced in the first week of February, completing on 31st March 2023, with remediation testing included.

We have received the report, and we understand the importance of reviewing and taking the necessary actions to address any vulnerabilities identified. We hope that the community finds this information helpful and reassuring, as we remain committed to ensuring the safety and security of all our users.

Comment 26

[Ahmed]

Attachment: E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş. Internet Sızma Testi Doğrulama Yönetici Özeti.pdf

Penetration Testing Report

Comment 27

[ryan_hurst]

Ahmed,

If I understand your response correctly you are saying the reason so much time has passed is you have been trying to coordinate with the vendors availability.

If so can you explain why the incident report provided stated:

Nov 18, 2022 19:55 : Starting third party penetration test action with full coverage

Also while I can appreciate the desire to keep sensitive detail’s confidential the scope of the penetration testing engagement is in essence two short sentences and provides no real detail on the scope of the engagement and what has been going on since November 18, 2022, or any real material detail one can use to asses what was done or what was found.

After waiting almost a half of a year for some meaningful response this is quite the letdown.

Comment 28

[Ahmed]

Hi Ryan

Let me clarify the timeline and the details of the penetration testing engagement to address your concerns. We kick started the third-party penetration testing process on November 18, 2022. However, as mentioned in the previous message, during the course of the testing.
We decided to further enhance the scope of the engagement to ensure comprehensive testing based on the reported incident. This decision required additional coordination with the testing team causing the delay in the actual testing cycle. Timely availability of the testing team remained another issue.

The actual testing commenced in the first week of February 2023 and was completed on March 31, 2023 as mentioned in my previous comment. Regarding the scope of the engagement, I understand your concerns about the limited information provided here. I thought it would not add any benefit announcing each step here. It’s also not possible to disclose the full details. I can assure you that the testing covered all relevant aspects of our system, including infrastructure, applications, network and internet security. The primary goal of the engagement was to identify potential vulnerabilities and ensure the robustness of our security measures.

I appreciate your patience. We take the security of our systems and the trust of our clients very seriously and we strive to maintain open communication throughout this process.

Comment 29

[Ben Wilson]

I have read through the executive summary, and I agree with Ryan, it is rather sparse. However, to eventually close this out, I think we need a report on the status of the remediation of the 4 medium and 9 low-level security vulnerabilities that were detected. When will those be fully patched? Is additional development, system re-engineering, or third-party software needed?

Comment 30

[ryan_hurst]

I have several concerns about the quality, consistency, and comprehensiveness of the incident report and information shared by e-Turga.

The report is not consistent and contradicts some provable facts from the initial reporter, raising questions about the reliability of the information provided.

e-Turga has selectively answered questions posed in the bug, leaving critical queries unaddressed, such as how basic security issues were overlooked during annual testing. This omission leaves many unanswered concerns.

We waited nearly half a year for the penetration testing report, which lacks essential information about the scope of the engagement and the activities performed.

e-Turga has not provided any information about the fortifications made as a result of their incident response. Reports are expected to include this level of detail. Without it, the root programs and the community cannot assess if similar issues are likely to reoccur in the future.

The incident report does not mention any lessons learned, suggesting that e-Turga's leadership views the situation as a mere inconvenience and fails to recognize its significance.

It appears e-Turga has not reviewed the expectations demonstrated in past incident reports. Comparing this report to those from DigiCert, Let's Encrypt, or Google Trust Services reveals a significant quality difference. Is this the standard Mozilla now expects for future reports?

The duration associated with handling this incident is also concerning. Comparing e-Turga's response to how DigiCert, Let's Encrypt, or Google Trust Services have handled incidents reveals a stark contrast, even for less severe issues.

The type of incident indicates e-Turga's failure to incorporate basic security practices into its development and deployment processes. This is especially concerning given the value a compromised CA represents to an attacker and the global consequences of a WebPKI CA compromise.

While regional CAs play an important role in the WebPKI, the nature of the system means that a failure in one CA puts the entire Internet at risk. All CAs must be held to the highest standards, but it seems that this is not currently the case, which is greatly concerning.

Moreover, there is the question of precedent: is this how we expect future incidents to be handled?

I believe it is crucial for Mozilla and other root programs to require a more thorough and comprehensive response from e-Turga. We need a clear understanding of the actions taken to address the incident, the changes implemented to prevent future occurrences, and the lessons learned from this experience if e-Turga is to remain in the root program.

Comment 31

[ryan_hurst]

It looks like spell check edited e-Tugra to e-Turga and I missed it before posting. Please accept my apologies.

Comment 32

[Hasan Berkay Cagir]

As a native Turkish speaker, I have to add that I understand almost nothing from the report. Only indicative looking part of the report is the penultimate paragraph, but even that paragraph does not tell anything substantial. I have been following this incident for a while now and it sadly has been nothing of vague statements that demonstrate nothing.

Comment 33

[Chris Clements]

We (Chrome Root Program), like others, were anticipating output from the penetration test activities that would substantiate many of the statements previously made in this incident report. However, the report provided in Comment #26 lacks actionable detail.

We have two requests related to the penetration test report that was provided and additional questions that need to be addressed.

We encourage e-Tugra to provide complete and timely responses to each of the requests and questions in this comment, as we seek demonstrable evidence for continued trust in e-Tugra as a publicly trusted CA.

---

Related to the Penetration Test Report

Request #1: Can an authoritative, English language version of the report be provided?

Request #2: Comment #5 describes general assessor qualifications, but is not clear whether those qualifications were illustrative or representative of the individual’s who authored the report added in Comment #26.

Please provide a description of the team and corresponding qualifications who authored the report to minimally include:

• Name of the lead assessor (unless prohibited by law)
• Number of team members
• Assessor academic qualifications or professional training received
• Years of assessor experience evaluating similar information systems
• Assessor experience, special skills, and qualifications (e.g., audit/assessment principles and functions, information technology, software development, trust services, public key infrastructure, CA operations, and information security including risk assessment/management, network security, physical security, etc.)
• Assessor credentials, designations, or certifications (e.g., CPA, CISA, CITP, CISSP, CCSP/CCA/CCP, etc.)
• How are the team members bound by law, regulation or professional standards to render an independent assessment?
• Whether the evaluation team relied on any third-party specialists or affiliate firms, and if so, their names and where they performed services.

---

Additional Questions

We feel there are several details related to the incident that are unclear and need to be more directly addressed.

Based on the contents of this report and the initial incident disclosure, it appears possible that unauthorized access to the “internal application” (i.e., accessed by Ian) could have:

• revealed domain control validation information (“leaked” random values via email messages observed in the “internal application”),
• resulted in certificate mis-issuance (where if an unauthorized third party observed “customer portal” password reset messages via email messages observed in the “internal application” they could have ultimately accessed the “customer portal” to perform certificate management functions (Comment #15), and
• revealed documents containing PII (e.g., citizen IDs, phone numbers, and physical addresses).

Questions:

1. The CPS that corresponds with the CAs responsible for issuing certificates identified in Comment #9 is owned by SSL.com. The CPS describes the opportunity for requesters to perform domain validation authorization via email. Was email-based domain validation an option for e-Tugra customers during the period in which the “internal application” was internet-accessible?

   • If the answer is “No,” please describe evidence that can be presented to a qualified auditor to corroborate that claim.

   • If the answer is “Yes,” please describe whether emails containing domain validation information (i.e., “random values”) could have been accessed by someone with access to the “internal application” (as demonstrated by Ian’s blog). To be specific, this is not asking whether you can confirm whether emails containing domain validation information were accessed, but whether it was theoretically possible for someone to access that information.

2. Comment #12 states “e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates but the accessed application does not have any impact in this process.” Would it have been possible for someone with access to the “internal application” (i.e., “accessed application”) to observe password reset emails for accounts with access to the “customer portal”, and use the observed reset information to gain access to the “customer portal”?

   • If the answer is “No,” please describe the controls in place that would have prevented this. Also, describe evidence that can be presented to a qualified auditor to corroborate that claim.

3. If the answer to question 2 (directly above) is “Yes,” please answer the following three questions (3A - 3C). If the answer to any of the questions is “No,” please describe the controls in place that would have prevented these activities from taking place. Also, describe evidence that can be presented to a qualified auditor to corroborate those claims.

   • 3A. Would it have been possible for them (i.e., someone who observed a “customer portal” account password via the “internal application”) to have used the “customer portal” to successfully perform certificate re-issuance for a domain without re-performing the domain validation process (i.e., assumes the domain successfully completed the domain control validation process less than 398 days prior (i.e., had “fresh” validation information)?

   • 3B. Would it have been possible for them to have used the “customer portal” to successfully perform certificate revocation?

   • 3C. Would it have been possible for them to have used the “internal application” or “customer portal” to make administrative access changes that could have either modified existing authorized “customer portal” user access or granted access permissions to new users?

4. Given the cross-certificate relationship with SSL.com, describe in detail SSL.com’s involvement in routinely evaluating e-Tugra’s infrastructure (minimally describe the “internal application” accessed by Ian), and the practices, policies, and processes related to certificate issuance and management.

5. Back in December (Comment #9), we asked if we could expect a discussion and corresponding evaluation of the “connection” in question in the penetration test (i.e., the connection between the internal application and issuing CA). The response was: “We have enhanced the scope to both CA and non-CA systems in the penetration testing exercise. Penetration testing firm will be evaluating both systems and identifying any gaps if any.” (Comment #10)

   We have reviewed the report uploaded in Comment #26 and do not see discussion related to the connection to the issuing CA as expected given the above statement. Is there specific discussion from the penetration test team regarding the connection in question that can be shared?

6. Is e-Tugra planning to provide an updated or supplemental report from the penetration test team that includes a more explicit narrative of what was in scope and out of scope for the penetration test? It is not clear based on the report added to Comment #26).

7. Is e-Tugra planning to provide an updated or supplemental report from the penetration test team that explicitly confirms the application(s) and corresponding systems described at https://ian.sh/etugra were included in the scope of the assessment?

8. Is e-Tugra planning to provide an updated or supplemental report from the penetration test team that details the testing approach (i.e., methods and corresponding timelines) and tools used to evaluate e-Tugra’s infrastructure?

9. Is e-Tugra planning to provide an updated or supplemental report from the penetration test team offering more detail related to the 13 vulnerabilities identified by the testing team? The table included in the report offers little insight or actionable detail. Reminder: Please DO NOT upload sensitive or proprietary information to this incident report if it puts user security at risk.

10. Is e-Tugra planning to provide more detail regarding the improvements made to its environment, to include process and technology, in a manner that other CAs can understand and consider applying to their own environment?

If the answers to each of the questions 5-10 (directly above) are “Yes,” when can we expect that output to be posted here? Ideally, that output is also provided in an authoritative, English language version.

Comment 34

[Thomas Zermeno]

Just to add clarification to Question 4., SSL.com operates a managed SubCA issued to E-Tugra, and we control all aspects of validation and issuance within our facilities. There is no cross-certificate relationship between SSL.com and E-Tugra.

Comment 35

[Ahmed]

Hi All,

Before providing the answers of the asked questions, let us clarify our domain validation and certificate issuance processes again as it looks we could not explain well because of our bad English (we are not native English speakers). During the period in which the "internal application" was internet-accessible, E-Tugra was not using an internal sub-CA to issue certificates. Instead, we partnered with an external CA authority (SSL.com) to ensure a smooth transition and continuing our business as one of Root CA hierarchy was about to expire and our new Root CAs hierarchies were in the inclusion process and we planned to use external managed subCA unless our new Root CAs are included in all root programs. SSL.com operates a managed SubCA for E-Tugra to handle all aspects of validation and issuance within SSL.com facilities.

We ensured that our domain validation process was secure and adhered to industry best practices. The managed SubCA is responsible for performing domain validation using approved methods and issuing certificates from their secure facilities. E-Tugra does not have any control in the validation process as a result, the potential risks associated with email-based domain validation and the internet-accessible "internal application" were mitigated.

We explained in comment #10 & google groups that the accessed application stores the logs information (which contains emails as archived and other information) used for generating reports and business analysis.

To further substantiate the claim, we can provide the following evidence to a qualified auditor:
• A formal agreement between E-Tugra and SSL.com outlining the roles and responsibilities of each party, including the management and operation of the SubCA
• Documentation of the domain validation policies and procedures implemented by SSL.com, demonstrating that our email-based domain validation was not used during the specified period
• System logs and records from the managed SubCA, showing the domain validation methods employed and the issuance of certificates within their facilities. We only log and archive our requests for reporting purposes
• Although accessed application has removed as it was mistakenly published but we will simulate the same application to present it to qualified auditors to observe that no email-based domain validation performed by E-Tugra and also the archived emails/notifications existing there

@Ryan, we asked the pen testing company to share the executive summary report of the findings. We fully understand and appreciate community demand for the sufficient details and we have requested the pen testing company the updated report which provides enough information. For the avoidance of such incidents, we have taken the actions mentioned in comment #18.

@Chris, we are sharing feedback against your queries also.

Request #1: Can an authoritative, English language version of the report be provided?
[E-Tugra]: We have requested the pen testing company to provide an English report with some additional details about the findings. We will post the report here once we receive it.

Request #2: Comment #5 describes general assessor qualifications, but is not clear whether those qualifications were illustrative or representative of the individual’s who authored the report added in Comment #26.
[E-Tugra]: We have requested all the desired information, we will share once we receive them.

Questions:
Q1: Answer is No. We have explained the evidence above to be presented to the qualified auditor.
Q2: Answer is No. We explained above that the accessed application contains the archived emails or logging information. The codes etc. visible in the accessed application have validity of three minutes only or the same code can not be re-used.
Q3: Not applicable
Q4: SSL.com has managed SubCA for E-Tugra, comment# 34. There is no cross certification in place.

For other questions we have requested the pen testing company to provide the details. We will be sharing the details here upon receiving.

In the end, we have fixed all the findings reported by the pen testing company. Pen testing company is performing remediation testing. The remediation results will be provided to us by the end of next week as there are Ramazan /Eid public holidays in Turkiye. We will share the report here (without sensitive information). That report will also be presented to the auditors.

Comment 36

[Chris Clements]

Ahmed, thank you for providing some clarification. We understand the language challenge and appreciate the timely response. We do have additional requests for clarification, as we want to continue to make sure our understanding is accurate.

--

SSL.com operates a managed SubCA for E-Tugra to handle all aspects of validation and issuance within SSL.com facilities.

Request for Clarification #1: SSL.com has issued more than one CA certificate with E-Tugra represented in the CN. In addition to your statement, Comment #34 also implies only a single subordinate CA exists, yet we can see multiple entries (below). To clarify, confirm “SSL.com operates a managed SubCA for E-Tugra” should instead be stated as “SSL.com operates multiple managed SubCAs for E-Tugra.”

1. CN=E-Tugra EV TLS ECC CA R1; O=E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI; C=TR

   Child CA(s):
   E-Tugra EV TLS ECC SubCA R1

2. CN=E-Tugra EV TLS RSA CA R1; O=E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI; C=TR

   Child CA(s):
   E-Tugra EV TLS RSA SubCA R1

3. CN=E-Tugra TLS ECC CA R1; O=E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI; C=TR

   Child CA(s):
   TrustSafe TLS ECC SubCA R1
   E-Tugra TLS ECC SubCA R1

4. CN=E-Tugra TLS RSA CA R1; O=E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI; C=TR

   Child CA(s):
   TrustSafe TLS RSA SubCA R1
   E-Tugra TLS RSA SubCA R1

All of the above CAs have records in the CCADB indicating SSL.com as the CA Owner.

--

We ensured that our domain validation process was secure and adhered to industry best practices. The managed SubCA is responsible for performing domain validation using approved methods and issuing certificates from their secure facilities. E-Tugra does not have any control in the validation process as a result, the potential risks associated with email-based domain validation and the internet-accessible "internal application" were mitigated.

Request for Clarification #2: Confirm that e-Tugra systems, to include the "internal application" accessed by Ian never had access to or stored domain validation “Random Values” or other challenges related to the domain validation process - regardless of whether:

1. they were generated by SSL.com, or
2. if the verification of the challenges related to those values was ultimately validated by SSL.com.

--

We explained above that the accessed application contains the archived emails or logging information. The codes etc. visible in the accessed application have validity of three minutes only or the same code can not be re-used.

Request for Clarification #3: Despite the short lifetime and/or prohibited reuse of password reset codes making it possibly more challenging for a bad actor to attack this system, it is not clear to us what would have prevented an unauthorized individual with access to the “internal application” from observing a customer portal password reset code, using that code to reset a valid customer portal user’s account password, and then gaining access to the customer portal. Please explain why it was impossible for someone with a password reset code to perform a password reset for an account they did not own (i.e., account takeover).

--
Request for Clarification #4: Provide an updated incident timeline that includes dates and times for:

• When the “internal application” was made publicly accessible.
• When security controls were corrected such that the “internal application” was no longer publicly accessible.
• The penetration test activities that resulted from the disclosure of this incident.
• All follow-up items and remediation actions related to the penetration test and corresponding findings.

--
Request for Clarification #5: Due to e-Tugra’s relationship with SSL.com, and as a matter of being included in multiple public root stores, there are multiple audit firms involved (e.g., BDO through the SSL.com managed SubCA(s) and LSTI through e-Tugra's own publicly trusted roots). The responses in Comment #35 indicate corroborating materials will be presented to a qualified auditor. Please clarify:

1. Which auditor(s) will be presented with the evidence described above?
2. How and when root programs and other interested parties should look to verify these commitments took place as described?

Comment 37

[Ahmed]

Hi Chris,

Please see e-Tugra feedback against each point:

Request for Clarification #1: SSL.com has issued more than one CA certificate with E-Tugra represented in the CN. In addition to your statement, Comment #34 also implies only a single subordinate CA exists, yet we can see multiple entries (below). To clarify, confirm “SSL.com operates a managed SubCA for E-Tugra” should instead be stated as “SSL.com operates multiple managed SubCAs for E-Tugra.”
[e-Tugra]: Yes there are multiple SubCAs managed by SSL.com for different certificate types EV SSL, OV and DV with key types e.g. RSA & ECDSA

Request for Clarification #2: Confirm that e-Tugra systems, to include the "internal application" accessed by Ian never had access to or stored domain validation “Random Values” or other challenges related to the domain validation process - regardless of whether:

1. they were generated by SSL.com, or
2. if the verification of the challenges related to those values was ultimately validated by SSL.com.
   [e-Tugra]: Yes the “random values” generated and other challenges for domain validation are performed by SSL.com even the validation emails are sent by SSL.com directly to customers.

Request for Clarification #3: Despite the short lifetime and/or prohibited reuse of password reset codes making it possibly more challenging for a bad actor to attack this system, it is not clear to us what would have prevented an unauthorized individual with access to the “internal application” from observing a customer portal password reset code, using that code to reset a valid customer portal user’s account password, and then gaining access to the customer portal. Please explain why it was impossible for someone with a password reset code to perform a password reset for an account they did not own (i.e., account takeover).
[e-Tugra]: As explained in comment 35 that codes etc. have limited validity so the internal application displays them once they are expired or invalid.

Request for Clarification #4: Provide an updated incident timeline that includes dates and times for:
When the “internal application” was made publicly accessible.
When security controls were corrected such that the “internal application” was no longer publicly accessible.
The penetration test activities that resulted from the disclosure of this incident.
All follow-up items and remediation actions related to the penetration test and corresponding findings.
[e-Tugra]: We will be publishing the updated incident report.

Request for Clarification #5: Due to e-Tugra’s relationship with SSL.com, and as a matter of being included in multiple public root stores, there are multiple audit firms involved (e.g., BDO through the SSL.com managed SubCA(s) and LSTI through e-Tugra's own publicly trusted roots). The responses in Comment #35 indicate corroborating materials will be presented to a qualified auditor. Please clarify:
Which auditor(s) will be presented with the evidence described above?
[e-Tugra]: e-Tugra will present to the LSTI auditors all the evidence described.
How and when root programs and other interested parties should look to verify these commitments took place as described?
[e-Tugra]: We assume that an audit letter with details explaining that the auditors witnessed all the evidence for this incident and submit it to CCADB.

Comment 38

[ian]

As explained in comment 35 that codes etc. have limited validity so the internal application displays them once they are expired or invalid.

I believe it is highly unlikely e-Tugra has or had a control that prevents the email from being displayed in the internal application until the confirmation code is expired or used. I did not experience this in my testing, and e-Tugra has not provided any level of detail or assurance that this is true. Additionally, as visible in the internal application[1], the confirmation email was created and visible inside the internal application at "13-11-2022 13:05:53 +03:00". I received the confirmation code at this same time[2], at 2:05am Pacific time on the 13th. It would appear there is no delay in the emails being visible in the application.

I remain concerned that numerous statements made by e-Tugra in this incident bug are at best unsubstantiated guesses on how their own system works, and at worst are intentionally downplaying the impact of these issues.

[1] https://i.imgur.com/MB022oo.png
[2] https://i.imgur.com/T8KPiIv.png

Comment 39

[Ryan Dickson]

Hi Ian,

Thank you for your continued participation in this discussion.

Can you please help us better understand statements made in Comment 15?

The vulnerabilities I discovered allowed me to view the password reset emails sent to any customer. As a result, I could log into any customer account, via resetting their password. There is evidence for this in the screenshots of my post.

Request for clarification #1: Can you please confirm whether you successfully executed this attack (i.e., account takeover), or whether it was merely hypothetical based on the observation of reset data? Specifically, did you observe a password reset email and use the information therein to gain access to the customer portal, even if for an account you owned (i.e., [e-mail])? This image suggests the attack may have been possible, but it’s unclear whether it was successful in practice.

I was unable to purchase a certificate in my own account because your website did not accept American credit cards. This is not relevant to the vulnerabilities.

Request for clarification #2: Can you confirm you arrived at the payment collection step of the certificate issuance workflow after performing a successful customer portal account takeover (as described above)?

Thanks,
Ryan

Comment 40

[Ryan Dickson]

Hi Ahmed,

Can you please share:

1. When will e-Tugra publish the updated incident report described in Comment 37?
2. When can we expect e-Tugra to share more detail related to the penetration testing reports as described and committed to in Comment 35?

Thanks,
Ryan

Comment 41

[Ahmed]

Attachment: Updated_Incident-Report.pdf

Hi All,

Please find the attached updated incident report. The pen testing company will be compiling the detailed pen testing reports covering remediation testing also. We are expecting to get the reports by the end of next week and will be posting here.

Comment 42

[ian]

(In reply to Ryan Dickson from comment #39)

Hi Ian,

Thank you for your continued participation in this discussion.

Can you please help us better understand statements made in Comment 15?

The vulnerabilities I discovered allowed me to view the password reset emails sent to any customer. As a result, I could log into any customer account, via resetting their password. There is evidence for this in the screenshots of my post.

Request for clarification #1: Can you please confirm whether you successfully executed this attack (i.e., account takeover), or whether it was merely hypothetical based on the observation of reset data? Specifically, did you observe a password reset email and use the information therein to gain access to the customer portal, even if for an account you owned (i.e., [e-mail])? This image suggests the attack may have been possible, but it’s unclear whether it was successful in practice.

Hi Ryan,

I do not believe I executed this attack in a controlled manner (i.e. I don't believe I specifically tried to "take over" my account). However, during my testing, I deliberately requested codes from the client application and then viewed them in the internal application, and I do not recall any delay in this process. There are really three separate account takeover issues in the E-Tugra panel that would be good to enumerate:

• The main issue is account takeover via viewing the password reset code sent to a user's email address. Anyone could go initiate this flow for any user, then use the internal application to view the correct code. The attacker could then complete the flow for any account.
• The E-Tugra panel also has a separate login flow called "quick login" (Hızlı Giriş Yapın), where a user can login via receiving a SMS code sent to their phone. SMS codes and messages were also visible in the internal application[1], as you can see via the "SMS" tab in the photo you linked. I do not recall any delay here either, but I do not believe this function worked with my own phone number.
• There were also several ways to perform an account takeover even without internal application access. For example, the codes in both of the above flows were simply four digit numbers, and there was no apparent rate limit on trying to guess the right code. Additionally, I reported a separate vulnerability to E-Tugra where an attacker could login to any account without any password or authentication so long as they knew that account's ID. I did not write about this explicitly at the time as I was not confident they had fixed it.

[1] I happened to find a screenshot of the SMS portal in the application, which I have uploaded at https://i.imgur.com/vSW3oz4.png.

Request for clarification #2: Can you confirm you arrived at the payment collection step of the certificate issuance workflow after performing a successful customer portal account takeover (as described above)?

With the caveats I explained in my first answer included, I never experienced any kind of "step-up" authentication or MFA prompts within the E-Tugra application. As a result I believe anyone who successfully takes over an E-Tugra panel account would have full control of the account without limitation.

Comment 43

[Ryan Dickson]

@Ian, thanks for the added clarification.

@Ahmed, three (3) questions below:

Question 1: Can you please clarify which of the following ICAs were integrated with the e-Tugra Customer Portal at any point when the log server (i.e., “internal application”) was internet-accessible (October 10, 2022 - November 14, 2022)? Specifically, in this case, “integrated” refers to any connectivity between the portal and the CA to facilitate certificate issuance and management functions. If any additional publicly-trusted ICAs were integrated with the Customer Portal and are not listed below, please share a corresponding crt.sh link.

1. E-Tugra EV TLS ECC SubCA R1
2. E-Tugra EV TLS RSA SubCA R1
3. E-Tugra TLS ECC SubCA R1
4. TrustSafe TLS ECC SubCA R1
5. E-Tugra TLS RSA SubCA R1
6. TrustSafe TLS RSA SubCA R1
7. E-Tugra Domain Validated CA ECC v3
8. E-Tugra Extended Validated CA ECC v3
9. E-Tugra Organization Validated CA ECC v3
10. E-Tugra Domain Validated CA RSA v3
11. E-Tugra Extended Validated CA RSA v3
12. E-Tugra Organization Validated CA RSA v3

Minimally, we observe #5 and #6 from images on Ian’s blog (described in Comment 9).

---

Question 2: Assuming an individual had access to the Customer Portal at any point when the log server (i.e., “internal application”) was internet-accessible, what certificate issuance and management privileges were within the scope of that Customer Portal access? Minimally, from Comment 12 (snippet below), we understand certificates with unexpired domain validation documentation could be re-issued without requiring re-completion of the domain validation process.

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates but the accessed application does not have any impact in this process.

---

Question 3: Assuming certificate issuance and management functions were possible through the Customer Portal (and described in response to Question 2, above), can you describe the high-level steps an authorized user would perform to successfully complete each of the following scenarios to help us understand how the portal works?

• Scenario A: A new user registers an account on the Customer Portal and requests a certificate for a new domain (not known to e-Tugra or SSL.com prior to certificate issuance, meaning domain validation re-use is not possible).
• Scenario B: An authorized Customer Portal user wants to re-issue a soon-to-be-expiring certificate. Assume domain validation was last successfully completed <30 days ago and that validation data can be re-used (i.e., it is not required to re-perform domain validation).
• Scenario C: An authorized Customer Portal user wants to revoke an active certificate for which their user account is authorized to act on.

If any of these questions are unclear or could benefit from additional explanation, please let me know.

Thanks,
Ryan

Comment 44

[Ryan Dickson]

This incident continues to highlight behavior in violation of the Chrome Root Program policy.

We have observed the following:

1. A failure to provide timely communications.

• This incident report has gone “stale” on multiple occasions where 1) the bug lingered for more than 7 days without an update while a nextUpdate was not set in the future, or 2) a question was not acknowledged or responded to within 7 days.
• At one point, the report went without an update for 26 days.

2. A failure to transparently describe architecture, implementation, operations, and external dependencies.

• The initial set of corresponding CAs potentially affected by this incident was only revealed after manually “scraping” dnsNames from screenshots posted to the security researcher’s blog and evaluating them against crt.sh.
• Clarification for the complete set of subordinate CAs integrated with the “Customer Portal” was requested in Comment 43 and remains unanswered (now “stale”).
• At many points throughout this bug, responses from e-Tugra failed to demonstrate either a commitment to or understanding of core security principles (e.g., inadvertently connecting a privileged internal application to the internet, permitting the use of default admin credentials, and Comment 35 indicating that bad actors could not have abused user account reset codes because they expire after 3 minutes).
• Related, we observed a lack of understanding of actual system design (e.g., Comment 5 describes no connection between the affected systems and the CA, but later Comment 10 suggests that a “one-way” connection exists, and Comment 38 presents contradictory evidence that further suggests statements in Comment 35 related to the transmission and visibility of user account reset codes were inaccurate).

3. A failure to demonstrate an understanding of the root causes of the incident.

• Incident root cause has not yet been clearly addressed.

4. A failure to demonstrate an understanding of the security impact of the incident.

• e-Tugra has failed to accurately convey or refute the potential security impact of this incident. For example, the following is translated from an announcement made on the e-Tugra website “In our investigations, it has been observed that no personal or valuable data has been collected and the accounts of any of our customers have not been accessed. A screenshot of the document belonging to the preliminary application form of only one certificate holder was taken. Except for the aforementioned document, no critical data, access rights, etc. belonging to our customers. There was no information leak.” We observed the researcher present contradictory claims in that he accessed three customer documents, and highlighted over 250,000 documents would have been available to unauthorized third parties while the e-Tugra Panel application was both 1) internet connected and 2) relying on default credentials with administrative privileges.
• Comment 1 describes that no certificates were affected by this incident, but it remains unclear what would have prevented a bad actor with access to the Customer Portal from renewing a certificate where the domain validation process was completed within the last 398 days or revoking a valid certificate for which they were not authorized to act upon.

The above set of concerns introduces doubt in the reliability of the information provided by e-Tugra, and fails to demonstrate why continued trust is justified. The Chrome Root Program continues to monitor this incident and will take the necessary action to protect the security interests of Chrome’s users.

Further, Comment 9 identified some issuing CAs corresponding with this incident are only publicly-trusted because of SSL.com root CAs.

The Chrome Root Program Policy states:

“CA owners with self-signed root CA certificates included in the Chrome Root Store must satisfy the requirements defined in this policy, including taking responsibility for ensuring the continued compliance of all corresponding subordinate CAs and delegated third parties participating in the Public Key Infrastructure (PKI).”

A separate report will be opened to discuss and better understand 1) SSL.com’s role in this incident and 2) their responsibility to ensure e-Tugra’s continued compliance with the Chrome Root Program policy.

Thanks,
Ryan

Comment 45

[Ahmed]

Attachment: E-Tugra-Executive-Summary-Test-Verification-Report.pdf

Hi All,

We are delighted to announce the successful completion of the penetration testing of our systems, conducted by a reputable security firm. The exhaustive tests have been concluded, and we have received the comprehensive report highlighting the findings.

While the executive summary provides a broad overview of the investigation, the detailed reports contain specific findings and the results of their validation. Due to the sensitive nature of this information, a complete disclosure is not feasible in this context. However, we assure you that these reports will be thoroughly presented and scrutinized during the audit process.

For the sake of transparency and community interest, we can share the following issues:

• Web Server / Application Server information disclosure
• Outdated versions of certain third-party applications
• Cross-Origin Resource Sharing configuration issues
• File uploading vulnerability on the support portal
• Usage of default error messages

We would like to address @Ryan's concerns about potential violations of the root program policy. We assure you that we have worked responsibly, focusing all our efforts to ensure the integrity of this task. Our primary aim was to provide valuable insights to the community, and not just adhere to policy compliance.

In particular, the remediation of outdated third-party applications was a complex task as it involved identifying suitable and more secure replacements. This process required time and led to a delay in our updates. We sincerely apologize for any misunderstanding that might have arisen due to language barriers and led to excessive community comments.

We remain committed to transparency and the sharing of information with the community and the root programs. The auditors will have complete access to all the required information and their conclusions will be included in the audit report for the community's review.

We are deeply grateful for everyone's patience and understanding during this process and we apologize for any delays experienced in reaching this final closure.

Comment 46

[xiaohui.lam]

@Ian,

I'm not a mozilla or chrome staff but a PKI interested practitioners.
But when I check your report of security concern with e-Tugra, I have few quandary or doubt.

• First, In this screenshot 568k sent emails from e-Tugra systems, the row contains About the order no 1049452..., 1-Complete Order Email in Case of Offer... at the list, have you look into and checked it?
• And second, did you reported the screenshot including domain control validation via email method mail subject & mail body? but i can't view any screenshot shows current mail title is About the order no 1049452... which same to the list. I saw a screenshot and it is surely the e-tugra system validation code which not involved to SSL.com, and another screenshot it surely is a template view, the validation code was passed by the {verificationkey} and can't proof SSL.com gives the random value to e-Tugra.

If there is no security incident exposed about SSL.com, please clarify these informations as are critical to mozilla pkipolicy, to another involving CA issuing certificates should be investigated further or not.

Thanks.

Comment 47

[ian]

I never saw anything that indicated E-Tugra does or doesn't have control over DCV for SSL.com-issued certificates, so I wouldn't be able to assert anything there.

As I mentioned in the post, the DCV part of the application appeared to be unused when I saw it, but I was not sure what certificates may have been issued via it in the past:

It appears this template may be presently unused by e-Tugra at the moment, however any domain validation previously completed through this system likely should be considered invalid.

Of course, E-Tugra delegating the DCV to SSL.com does not help if the compromised E-Tugra customer portal can reissue existing certificates without going through DCV again.

Comment 48

[xiaohui.lam]

thanks to Ian’s information.

Ok, so the community now aren’t sure how 4 domains validation being proceeded in comment 9,
I think we need SSL.com to provide the report or log of how validation takes. To proof

• ssl.con sents email random tokens to domain owner directly, without telling e-Tugra. OR
• subscribers aren’t asking for the email validation method. (So the concern in Ian’s blog “Email validation is the only allowed domain validation method that relies on “secret” values” exists no more).

I think the validation log is the strongest proof of SSL.com’s system and SSL.com’ partnership w/ e-Tugra vulnerable or not.
If their system is strongly secured and partnership has no security concerns(such as RA grants outside), i think it should continue to be trusted for e-Tugra’s subCAs under SSL.com, but important premises: they fixed all security incidents and provided detailed, reliable, third security tests. In historic there are some CAs were removed from trust store but continue to resell certificates under another CAs’ subordinate.(eg: wosign).

Comment 49

[Ben Wilson]

Attachment: PSYFhqV.png

Saving images for future reference

Comment 50

[Ben Wilson]

Attachment: Screenshot_2022-11-13_at_3.40.47_AM.png

Comment 51

[Ben Wilson]

Attachment: Screenshot_2022-11-13_at_3.41.27_AM.png

Comment 52

[Ben Wilson]

Attachment: Screenshot_2022-11-13_at_3.47.55_AM.png

Comment 53

[Ahmed]

Hi Root Store team and Community members,

We hope this comment finds you well.

We identified that there are some comments which are unattended and we wanted to share the feedback for them.

Comment 43, @Ryan please see feedback on your queries:

[e-Tugra]: CAs from 1 to 6 are sub-CAs managed by SSL.com for e-Tugra handling the certificate life cycle and especially domain validation emails are directly handled by SSL.com to subscribers directly. CAs from 7 to 12 are sub-CAs of e-Tugra Root hierarchies and are not engaged for certificate life cycle activities because these are not fully included in all root programs. For this incident, there is no CA integrated with the exposed application to impact the certificate issuance. As explained above, the template shown in the screenshot was not used for the domain validation process.
As mentioned in comment 12, portal allows the user to reissue certificates for non-expired domains after performing the authentication but the exploited application cannot be helpful to gain privileged access to user credentials because it records the expired codes etc. as archived data.
An authorized user can request for a new domain certificate. The application is sent to SSL.com as explained above, all the domain validation checks are performed directly by SSL.com. There is no involvement from e-Tugra in the domain validation steps.
An authorized user can request to re-issue a certificate for already validated domain information according to CA/B guidelines
An authorized user can request to revoke an active certificate. Once submitted, an operator receives the revocation request and approves/declines the request for certificate revocation.

Comment 44, @Ryan please see our feedback:
[e-Tugra]: We acknowledge that this incident marked our initial encounter, and we accept responsibility for our administrative performance here, which contributed to the identified shortcomings.

We acknowledge that the language barrier has caused confusion, leading to imprecise expression of our statements.

In subsequent comments, we made a concerted effort to provide additional details. Both exploited application and CA systems do not interact for the certificate life cycle operations and have their own boundary while the CA system pushes the logs/notifications after the execution of the operations using a secure flow channel to non-SSL system for archival and business decisions.
We have thoroughly investigated and attempted to address the root cause, documenting it in the incident report based on our understanding. We genuinely value your input and would greatly appreciate it if you could identify any specific points that would assist us in aligning our understanding with yours.

@xiaohui.lam @Ian, we have explained above that domain validation steps are directly performed by SSL.com for DV certificates. E-Tugra is not involved in this process. We have tried to address your points in Ryan’s queries also. In addition, we have already requested SSL.com to give information about it to the community.

@Ben, Regarding your comments numbered 49, 50, 51 & 52. We wish to highlight that we have previously provided feedback on these points in our previous comments. However, we are reiterating our feedback to ensure that none of these comments are overlooked. Our intention is to ensure that all concerns and suggestions receive the necessary attention and consideration they deserve. Please see our feedback:
· List of documents shown in the first image are related to subscribers and resellers. Those documents are not related to domain validation or domain certificates. Ian downloaded documents that were not related to SSL certificates directly.
· The code shown in the image cannot be used as it is valid for 3 minutes only and shown in the application as archived items.
· The third image displays the different operations logs and notifications e.g.; user activity logs, application requests logs etc. These are not related to SSL or certificate life cycle processes.
· The fourth image is the email template. This email template is not being used in sending email to subscribers in the domain validation process as SSL.com directly communicates with subscribers for domain validation.

In summary, it is important to note that the application exploited in this incident was not directly associated with the certificate life cycle process. Its primary purpose revolved around recording user activities, reseller activities, email notifications, and similar logging information. Rest assured, no certificates were affected by this incident, whether they belonged to e-Tugra CAs or SSL.com managed sub-CAs.
We promptly reported the incident through a Bugzilla bug, providing a comprehensive incident report. While we made concerted efforts to address community comments and provide feedback, we acknowledge that there were shortcomings in the execution of our administrative responsibilities in this thread. Unfortunately, this resulted in frustration and non-compliance with root store policies, which was highly regrettable and not in line with our intentions.

To ensure confidentiality, we have provided the requested details to the best of our understanding. Additionally, we have gathered substantial evidence of this incident for our auditors in the upcoming audit. They will thoroughly analyze the evidence and add relevant information into the audit report, reflecting the details surrounding this incident.

We remain committed to closely work and collaborate with all root stores, community members and are pleased to provide the requested information while ensuring confidentiality measures are in place.

We firmly believe that this comment will serve as a valuable contribution for Bugzilla admins, helping them in effectively resolving and closing this bug.