Open Bug 1801345 Opened 4 months ago Updated 9 hours ago

E-Tugra: Incident Report (Security Issues)

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: ahmed, Assigned: ahmed, NeedInfo)

Details

(Whiteboard: [ca-compliance] [uncategorized])

Attachments

(1 file)

Incident-Report.pdf
35.64 KB, application/pdf
Details
Attached file Incident-Report.pdf —

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0

1 . How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.
An email from Ian Carroll <ian@ian.sh> on Nov 13, 2022.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Nov 13, 2022: An email from Ian Carroll about finding
Nov 13, 2022: The issues were fixed immediately
Nov 14, 2022: We sent an email to the reporter for our progress.
Nov 14, 2022: Began to analyze both the impacts and reasons. All user activities are traced from the last one year.
Nov 17, 2022: Reporter published a blog about this issue.
Nov 18, 2022: Incident Report is published with this text with Mozilla Incident Report Format.
Our security team started work immediately on the reported issues along with a full scan of all the systems.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
No certificates are involved in the incident. CA continued to serve as expected.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
No certificates are involved in the incident. CA continued to serve as expected.
Customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition.
Only three documents related to some users' (non SSL) agreements are accessed by the reporter.
5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
No certificates are involved in the incident.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The information shared by the reporter is for an internal application. Restricted access policy was not applied well on the application so the issue is based on violation of access policy. Issue was fixed immediately and on a safe side complete system scanning activities were performed to ensure no other applications were affected. No sensitive information disclosure revealed in this issue.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
We have attempted remediation actions to overcome this issue as part of full system scanning. We have a process in place for the penetration and vulnerability assessment of the system which was executed on the planned time. In addition to another cycle of penetration/vulnerability exercise we are closing gaps in the access policy management process and audits.
We aim to execute these improvements by the end of this year.

Information about the incident is available at:
https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s

Thank you to Ian Carroll for identifying these concerns.

Ahmed, we have a few initial questions to help us better understand the scope and perceived impact of what is being reported. Additional questions will likely surface as more information becomes available.

  1. The timeline of actions in #2 must be updated to include timestamps, and should account for relevant events before the initial communication from the security researcher such as:

    • when each issue was introduced into the environment (answering the question “for how long as this vulnerability existed?”)
    • when the last penetration test, risk assessment, and qualified audit of the publicly-trusted hierarchy were each performed.

  2. The phrase “The information shared by the reporter is for an internal application.” could suggest that the system the security researcher accessed is completely separated from E-Tugra’s publicly-trusted hierarchies, i.e., the certificate hierarchies terminating at the following root CAs:

    CN=E-Tugra Certification Authority,OU=E-Tugra Sertifikasyon Merkezi,O=E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş.,L=Ankara,C=TR
    Certificate Hash: B0BFD52BB0D7D9BD92BF5D4DC13DA255C02C542F378365EA893911F55E55F23C
    SPKI Hash: c1ad1b1898ec395048df070bfa217e25c913bed8ca6b73de085528846a0103c1

    CN=E-Tugra Global Root CA ECC v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
    Certificate Hash: 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13
    SPKI Hash: 56f2ea8684106d0c73333951059d2bff9ace0a9934f015c5d84c5a959dfbb3cc

    CN=E-Tugra Global Root CA RSA v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
    Certificate Hash: EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2
    SPKI Hash: b3effbf46bcf66aedf71427e6bd60bf1a1878c7b72cab178703485fda6e3db38

    • Please confirm this interpretation is correct (i.e., what the researcher accessed is separate from the systems related to the hierarchies above).

  3. If the above interpretation is correct, describe, in detail, how the hierarchies stated above are separated from the applications and corresponding systems accessed by the security researcher.

    • Minimally describe whether the compromised systems and those related to the publicly-trusted hierarchies above are:
      • physically separated?
      • logically separated?
      • “air-gapped'' from one another, or otherwise segmented such that there’s no possible route between the two?

  4. Describe, in detail, the security controls in place that would have prevented an adversary from traversing systems/networks to gain access to systems or applications that correspond to E-Tugra’s publicly-trusted CA hierarchies. Provide sufficient detail such that we can objectively evaluate them as being satisfactory safeguards against the vulnerabilities described by the security researcher.

  5. Describe the maximum level of access or functional privilege an unauthorized third party could have exercised through the attack, especially concerning E-Tugra’s publicly-trusted CA systems or related PKI components (e.g., could an adversary use their access to request new certificates, intercept domain validation random values, perform certificate management functions, remotely manage network-attached HSMs, exfiltrate subscriber data, etc.?)

  6. Describe whether the vulnerabilities detailed by the security researcher also existed on E-Tugra’s publicly-trusted CA systems or related PKI components. What evidence does E-Tugra plan to present to their auditor in response to this incident to corroborate this statement?

  7. Describe the steps taken to evaluate whether the publicly-trusted hierarchies and corresponding systems capable of impacting certificate issuance and management are unaffected by the vulnerabilities described by the security researcher.

  8. How did you verify “customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition” for all publicly trusted TLS certificates issued by E-Tugra?

  9. In the remediation steps you stated another cycle of penetration/vulnerability exercises.

    • When will these exercises occur?
    • Who will perform the exercises and will they attest to the completeness of scope including all system elements that interact with E-Tugra’s publicly-trusted hierarchies?
    • What are the assessor’s qualifications?
    • What is the assessor’s independence?
    • How will the resulting report(s) be shared?
Assignee: nobody → ahmed
Type: defect → task
Summary: E-Tugra - Incident Report (Security Issues) → E-Tugra: Incident Report (Security Issues)
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]
Flags: needinfo?(ahmed)

Hi Chris Clements,

We are working with several internal teams to get their feedback and analysis on the problems being reported by researcher.

We are compiling a detailed report based on feedback from each team before we publish it. We hope that we will able to complete this activity within a week time.

Hi Chris,

Thanks for the queries. Before we give the detailed feedback on the questions, it is better to share some insight which will be surely helpful to digest the information against answers below. Both non-SSL applications and CA systems are physically and logically separated systems. We have dedicated teams for managing them. In order to access the CA system we have a proper dedicated physical M of N rule with multi-factor authentication applied. Same approach is applied on HSM access also. Even HSMs and CAs are also isolated. We do an annual CA external audit and our auditing company is a European company. Turkish audit company not selected intentionally to be biased or influenced. Every year a full physical site audit is performed by the auditor for the CA system. In our recent audit, auditors visited our site physically and they observed all the explained operations by full auditing CA system. Our CA logging has a signed timestamped and non-tamper system. Auditors reviewed our CA logging system and ensured it was non-tampered. This incident happened on another system as one of our internal application developed (tracking reseller pre-application activities) published a month ago about non-SSL business.

Please see inline-responses below against each query:

  1. The timeline of actions in #2 must be updated to include timestamps, and should account for relevant events before the initial communication from the security researcher such as:
    o when each issue was introduced into the environment (answering the question “for how long as this vulnerability existed?”)
    [e-Tugra]: Details of actions taken:
    ▪ Nov 13, 2022 15:11 (UTC +3): An email from Ian Carroll about finding
    ▪ Nov 13, 2022 15:26 (UTC +3): All teams in e-Tugra were notified and started work on the issue immediately
    ▪ Nov 13, 2022 15:44 (UTC +3): The issues were fixed immediately, by closing related sites
    ▪ Nov 14, 2022 09:50 (UTC +3): Additional email came from the reporter
    ▪ Nov 13-14, 2022: Our all team started to scan all systems and completed. The affected applications were closed.
    ▪ Nov 14, 2022 19:36 (UTC +3): We sent an email to the reporter for our progress.
    ▪ Nov 15, 2022:
    ● e-Tugra teams started to analyze both the impacts and reasons. All user activities were tracked
    ● We have executed a comprehensive exercise for all our services (both public and internal)
    ● All access rules and implementation are reviewed
    ● All applications and services are checked with cross control for the authentication and authorization rule
    ● Checked the CA system and didn’t find any impact for mis-issuance / reissuance of certificate, logging component etc.
    ▪ Nov 17, 2022 13:27 (UTC +3): Reporter sends notification about his blog.
    ▪ Nov 17, 2022 13:27 (UTC +3): Reporter published a blog about this issue.
    ▪ Nov 18, 2022 19:55 (UTC +3): Incident Report was published on Bugzilla.
    Our security team continued work on:
    ▪ Improvement on security control to prevent such issues
    ▪ Internal security test and scan, cross checks all access rules
    ▪ Starting third party penetration test action with full coverage
    ▪ Revising procedures and fix the gaps if any
    o when the last penetration test, risk assessment, and qualified audit of the publicly-trusted hierarchy were each performed.
    [e-Tugra]: CA system penetration testing was performed last year based on the recommendations from CA/B network security guidelines. Penetration testing report was presented to auditors like yearly audit. Risk assessment is also practiced annually or whenever there is a change that requires “risk review” in the system/procedures. We performed the planned activity this year and it was approved by the e-Tugra security board. We have gone through the ETSI TSP Audit in August 2022.

  2. The phrase “The information shared by the reporter is for an internal application.” could suggest that the system the security researcher accessed is completely separated from E-Tugra’s publicly-trusted hierarchies, i.e., the certificate hierarchies terminating at the following root CAs:
    CN=E-Tugra Certification Authority,OU=E-Tugra Sertifikasyon Merkezi,O=E-Tuğra EBG Bilişim Teknolojileri ve Hizmetleri A.Ş.,L=Ankara,C=TR
    Certificate Hash: B0BFD52BB0D7D9BD92BF5D4DC13DA255C02C542F378365EA893911F55E55F23C
    SPKI Hash: c1ad1b1898ec395048df070bfa217e25c913bed8ca6b73de085528846a0103c1
    CN=E-Tugra Global Root CA ECC v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
    Certificate Hash: 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13
    SPKI Hash: 56f2ea8684106d0c73333951059d2bff9ace0a9934f015c5d84c5a959dfbb3cc
    CN=E-Tugra Global Root CA RSA v3,OU=E-Tugra Trust Center,O=E-Tugra EBG A.S.,L=Ankara,C=TR
    Certificate Hash: EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2
    SPKI Hash: b3effbf46bcf66aedf71427e6bd60bf1a1878c7b72cab178703485fda6e3db38
    [e-tugra]: Already explained above, the CA system is both physically and logically separated.
    o Please confirm this interpretation is correct (i.e., what the researcher accessed is separate from the systems related to the hierarchies above).
    [e-Tugra]: We confirm that the affected system is separate from the CA system.

  3. If the above interpretation is correct, describe, in detail, how the hierarchies stated above are separated from the applications and corresponding systems accessed by the security researcher.
    o Minimally describe whether the compromised systems and those related to the publicly-trusted hierarchies above are:
    ▪ physically separated?
    ▪ logically separated?
    ▪ “air-gapped'' from one another, or otherwise segmented such that there’s no possible route between the two?
    [e-Tugra]: CA systems have separated physical locations that have high zone security physical and logical access controls. They have separate firewalls that apply very high restrictions and own log systems.

  4. Describe, in detail, the security controls in place that would have prevented an adversary from traversing systems/networks to gain access to systems or applications that correspond to E-Tugra’s publicly-trusted CA hierarchies. Provide sufficient detail such that we can objectively evaluate them as being satisfactory safeguards against the vulnerabilities described by the security researcher.
    [e-Tugra]: CA system is not exposed publicly and it has a separate network managed in a privately owned physical data center and even e-Tugra CA staff can not access any or traverse any machine in the CA system without physical M of N rule and multi-factor authentication principle in place. We have proper audit logs as well who entered into CA systems and logs are non-tampered.

  5. Describe the maximum level of access or functional privilege an unauthorized third party could have exercised through the attack, especially concerning E-Tugra’s publicly-trusted CA systems or related PKI components (e.g., could an adversary use their access to request new certificates, intercept domain validation random values, perform certificate management functions, remotely manage network-attached HSMs, exfiltrate subscriber data, etc.?)
    [e-Tugra]: As mentioned above, CA systems can not be accessed by unauthorized third parties. Remote access to network attached HSMs in the CA system is not possible because they are isolated. Security researcher attempted to get domain validation certificates multiple times but could not get it because of secure access control applied and he confirmed it in the blog. For domain validation values can only be accessed by trusted CA staff after authentication rules explained above. CAs, HSMs and related logs can’t be accessed remotely as it requires physical presence based on defined procedures above.

  6. Describe whether the vulnerabilities detailed by the security researcher also existed on E-Tugra’s publicly-trusted CA systems or related PKI components. What evidence does E-Tugra plan to present to their auditor in response to this incident to corroborate this statement?
    [e-Tugra]: Like every year, we will present all the sites, applications, network logs, system topologies, internal security controls and pen testing reports etc. to the auditor. Even auditors can ask anything related during auditing activities.

  7. Describe the steps taken to evaluate whether the publicly-trusted hierarchies and corresponding systems capable of impacting certificate issuance and management are unaffected by the vulnerabilities described by the security researcher.
    [e-Tugra]: We checked the CA system and all the non-tampered timestamped logs to find any attempt for the mis-issuance of the certificate or unauthorized access in the CA system.

  8. How did you verify “customers sensitive information e.g. credentials and similar information could not be obtained and are in safe condition” for all publicly trusted TLS certificates issued by E-Tugra?
    [e-Tugra]: Strong encryption is being applied to protect the customer's sensitive information for TLS certificates and no sensitive information is logged in the logging system.

  9. In the remediation steps you stated another cycle of penetration/vulnerability exercises.
    o When will these exercises occur?
    [e-Tugra]: We have already initiated the process with a third party penetration testing company. Once the plan is finalized we’ll share it here. We aim that it will be started as soon as possible.
    o Who will perform the exercises and will they attest to the completeness of scope including all system elements that interact with E-Tugra’s publicly-trusted hierarchies?
    [e-Tugra]: We already shared the complete requirements which covers the internal, external infrastructure, DDOS attacks, web application etc. penetration testing. Each area has its own rules & practices which will be performed as part of this exercise by third party pen testers.
    o What are the assessor’s qualifications?
    [e-Tugra]: Third party pen testers has got following certifications /skillset:
    ● Certified Information Systems Security Professional (CISSP)
    ● Offensive Security Certified Professional (OSCP)
    ● Certified Secure Software Lifecycle Professional (CSSLP)
    ● Certified Ethical Hacker (CEH)
    ● ElearnSecurity Web application Penetration Tester (eWPT)
    ● Practical Network Penetration Tester (PNPT)
    ● Offensive Security Wireless Professional (OSWP)
    ● Certified Secure Application Programmer (C-SAP)
    ● Microsoft Certified Solutions Associate (MSCA)
    ● Linux Professional Institute Certification (LPIC)
    o What is the assessor’s independence?
    [e-Tugra]: Penetration testing company is an independent third party. They will be executing the complete exercise.
    o How will the resulting report(s) be shared?
    [e-Tugra]: They will be sharing the penetration testing reports once they are done with the exercise. We’ll present reports to the auditors like we presented earlier.

Flags: needinfo?(ahmed)

I'd like to add some follow-up questions to those from Chris Clements in Comment #3 :

10. Regarding the timeline of events before the initial communication from the security researcher:

a. When were the prior vulnerability scans and penetration tests performed?

b. Did the scope of prior vulnerability scans and penetration tests include non-CA systems, given that advanced persistent threats often leverage office systems and use them to hop into CA systems? If no, then why not, and if yes, then which systems?

c. When was the last risk assessment performed and comprehensive security plan updated, both of which are required annually by Section 5 of the CA/Browser Forum’s Baseline Requirements?

11. Regarding the scope of e-Tuğra's upcoming vulnerability scans and penetration tests:

a. How will we know whether testing was adequately scoped to include systems that interact with e-Tuğra’s certificate-issuing systems, e.g. RA systems, customer portals, and other related non-CA components?

b. Will testing be white box, black box, gray box, or a combination of the three types?

c. What is the reference source or catalog of vulnerabilities that testing will examine?

12. Regarding the results from e-Tuğra's vulnerability scans and penetration tests:

a. How will we know that detected vulnerabilities have been remediated or that a remediation plan has been adopted and is being implemented? (We assume that you will be documenting the rationale for determining that a vulnerability did not require remediation, as required by section 4.f.3 of the Network and Certificate System Security Requirements.)

b. After remediating the vulnerabilities detected by the upcoming testing exercises, will you re-test for vulnerabilities and confirm that they have all been remediated?

Flags: needinfo?(ahmed)

Hi Ben,
Thanks for the comment. Please see [e-Tugra] feedback against each point below:

  1. Regarding the timeline of events before the initial communication from the security researcher:
    a. When were the prior vulnerability scans and penetration tests performed?
    [e-Tugra]: We do vulnerability scans monthly and penetration tests annually. We present the reports to auditors in the annual audit.
    We executed the vulnerability scan in October 2022 and Penetration Test in September 2022, before the current incident was reported. We have scheduled another cycle of penetration testing in the coming months. We will share the reports with auditors.

b. Did the scope of prior vulnerability scans and penetration tests include non-CA systems, given that advanced persistent threats often leverage office systems and use them to hop into CA systems? If no, then why not, and if yes, then which systems?
[e-Tugra]: Both CA and non-CA systems were in the scope of previous vulnerability and penetration exercise. The exploited application was published publicly a month ago prior to incident

c. When was the last risk assessment performed and comprehensive security plan updated, both of which are required annually by Section 5 of the CA/Browser Forum’s Baseline Requirements?
[e-Tugra]:We follow the CA/B baseline requirements section 5 for the annual risk assessment. Additionally we do whenever there is some change which requires risk assessment to be performed. We recently performed a risk assessment activity in February 2022.

  1. Regarding the scope of e-Tuğra's upcoming vulnerability scans and penetration tests:
    a. How will we know whether testing was adequately scoped to include systems that interact with e-Tuğra’s certificate-issuing systems, e.g. RA systems, customer portals, and other related non-CA components?
    [e-Tugra]: We can send the scope to the browser's program main contact personnel and will present the scope and reports to auditors in our next audit. We thought that sharing scope details publicly is not a good option.
    In this penetration and next all systems that we have CA and non-CA will be covered.

b. Will testing be white box, black box, gray box, or a combination of the three types?
[e-Tugra]: Yes, all three types will be covered in the upcoming testing scope.

c. What is the reference source or catalog of vulnerabilities that testing will examine?
[e-Tugra]: Penetration testing company has already the list of sources and catalog as part of their testing processes. We can share the catalog penetration testing company will use.

  1. Regarding the results from e-Tuğra's vulnerability scans and penetration tests:
    a. How will we know that detected vulnerabilities have been remediated or that a remediation plan has been adopted and is being implemented? (We assume that you will be documenting the rationale for determining that a vulnerability did not require remediation, as required by section 4.f.3 of the Network and Certificate System Security Requirements.)
    [e-Tugra]: Like previous vulnerability scans and penetration testing exercises, we will create a report against each vulnerability and present it to the auditor. These reports cover how and when they are fixed. If a remediation is not desired, the reasons and risks are reported. We are following the recommendations mentioned in the Vulnerability Detection and Patch Management section of Network and Certificate System Security Requirements.

b. After remediating the vulnerabilities detected by the upcoming testing exercises, will you re-test for vulnerabilities and confirm that they have all been remediated?
[e-Tugra]: Yes, the testing scope covers the re-validation of fixed remediation.

Flags: needinfo?(ahmed)

I'm setting the next update on this bug to January 6, 2023, with the expectation that e-Tuğra can provide an update on its penetration and vulnerability testing.

Whiteboard: [ca-compliance] → [ca-compliance] Next update 2023-01-06

Hi Ahmed,

While reviewing the images on Ian’s blog, I noticed what appeared to be dnsNames in this screenshot.

For example, in the “Title” column, we see the following dnsNames appear in what seem to be messages to subscribers confirming certificate issuance (i.e., the title contains the phrase "Your SSL Certificate Is Ready"):

In all of the above cases, the dnsNames correspond with a pre-certificate issued by either:

  • CN = TrustSafe TLS RSA SubCA R1, O = Isimtescil Bilisim A.S., C = TR, or
  • CN = E-Tugra TLS RSA SubCA R1, O = E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI, C = TR

Both of these issuing CAs chain to CN = E-Tugra TLS RSA CA R1, O = E-TUGRA EBG BILISIM TEKNOLOJILERI VE HIZMETLERI ANONIM SIRKETI, C = TR (1 and 2), and eventually CN=SSL.com Root Certification Authority RSA, O=SSL Corporation, L=Houston, ST=Texas, C=US, meaning these certificates are publicly-trusted.

Studying the timing of the entries in the image described above against the validity dates included in the pre-certificates listed above presents a pretty consistent delta between the time of pre-certificate issuance and customer notification (about 11 minutes).

Domain Crt.sh issuance time (GMT) Ian’s Blog Image Time (GMT +3) Ian’s Blog Image Time (Adjusted to GMT [-3]) Difference
bakbuonemli.com 11/13/2022 11:10:18 11/13/2022 14:21:32 11/13/2022 11:21:32 11 minutes and 14 seconds
*.kolaykvkk.com.tr 11/13/2022 08:22:46 11/13/2022 11:33:56 11/13/2022 08:33:56 11 minutes and 10 seconds
ankarabozkurtnakliyat.com 11/13/2022 08:00:11 11/13/2022 11:11:25 11/13/2022 08:11:25 11 minutes and 14 seconds
esperdetasarim.com 11/13/2022 07:11:43 11/13/2022 10:22:54 11/13/2022 07:22:54 11 minutes and 11 seconds

This timing consistency seems to suggest some degree of connectivity (e.g., an automated push) between the issuing CA and the system Ian accessed, which I interpret to mean the system described in Ian's blog post is not “separate” from the CA system (contradicts the following points made in Comment 5:

[e-tugra]: Already explained above, the CA system is both physically and logically separated.
and
[e-Tugra]: We confirm that the affected system is separate from the CA system.

A few questions:

  1. Is the above understanding correct? (i.e., there is some connection between the system Ian accessed and the issuing CA represented in the pre-certificates above that allows it to become aware of certificate issuance)
  2. Can we expect a discussion and corresponding evaluation of this connection in the penetration test?
  3. Comment 7 states:

[e-Tugra]: We do vulnerability scans monthly and penetration tests annually. We present the reports to auditors in the annual audit.
and
The exploited application was published publicly a month ago prior to incident.

However, it's not clear if "published" means "first deployed", "first made internet accessible", or something else. Can you clarify your intended use of "published"?

Thanks,
Ryan

Flags: needinfo?(ahmed)

Hi Ryan,

Thanks for your comment. Please see [e-Tugra] feedback against each point below:

A few questions:
Is the above understanding correct? (i.e., there is some connection between the system Ian accessed and the issuing CA represented in the pre-certificates above that allows it to become aware of certificate issuance)
[e-Tugra]: We stated earlier that the accessed application was intended for internal reporting usage. Once the CA system has performed its operations it then passes the post certificates specific log information using one-way flow secure network connection to the reporting system. The application classifies it based on defined rules to generate reports. It doesn’t have any direct operational connection with the CA system for the certificate life cycle process as domain validation emails are sent by the separated CA system and there is no automated way for the certificate issuance.

Can we expect a discussion and corresponding evaluation of this connection in the penetration test?
[e-Tugra]: We have enhanced the scope to both CA and non-CA systems in the penetration testing exercise. Penetration testing firm will be evaluating both systems and identifying any gaps if any.

However, it's not clear if "published" means "first deployed", "first made internet accessible", or something else. Can you clarify your intended use of "published"?
[e-Tugra]: We apologize for the inconvenience caused due to word selection. This application made the internet accessible for the first time mistakenly.

We stated earlier that the accessed application was intended for internal reporting usage. Once the CA system has performed its operations it then passes the post certificates specific log information using one-way flow secure network connection to the reporting system.

I am a little worried that we've gone from no connection to "one-way flow" connection, and I do not think that is true either. The emails in the administrative system that was accessed were related to the e-Tugra customer portal. The customer portal is intrinsically used to purchase certificates by customers, so orders must somehow flow from the customer portal to the issuance infrastructure, and then back again once issued. I do not see how it is possible for there to only be a "one-way flow".

As I stated on m.d.s.p., it is a common feature among CAs that customer portals can re-use existing validations, or re-issue existing certificates, without completing domain-control validation again. It would be good to know if e-Tugra's customer portal could do this, as it would have allowed an attacker to re-issue any existing e-Tugra certificate if so.

Hi Ian,

The reporting system receives the information from the CA system via some secure tunneling and generates emails/notifications not only for certificate related processes but also for other applications (resellers, internal applications etc.). This application does not participate or play any role in the certificate issuance process instead emails or reporting.

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates but the accessed application does not have any impact in this process.

Flags: needinfo?(ahmed)

Hi Ahmed,

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates but the accessed application does not have any impact in this process.

The e-Tugra customer portal allows resetting account passwords via email. The password reset codes were then displayed in the accessed application. As you can see in my post, there are emails of "Password Reset Request" visible in the application.

As a result, anyone who accessed the affected application could have logged into any e-Tugra account, and (it appears) then re-issued any e-Tugra certificate.

Hi Ian,

All DV validation and issuing certificate life cycles are processed by a separate system after request validation so this system does not hold any impact for the certificate life cycle.
We checked in our logging system and you attempted multiple times to place an order for the certificate through the portal but could not get the certificate. Now this shows that you want to get more CA system details to emphasize your point of interrupting the certificate life cycle process but it could not happen even with your attempts. We clearly state that there is no connection of the accessed application to the CA system for the certificate life cycle process.

Once this incident was reported we immediately reacted and reported on bugzilla (with detailed incident report) to inform related parties and even informed our customers about this incident and no DV certificate being re-issued as part of this incident. We thank you for reporting this incident.

We mentioned above that we are doing a comprehensive penetration testing and we will be sharing the penetration testing results with the auditors as we do it each year.

Hi Ahmed,

I am not sure what your comment is trying to refute. You previously said:

e-Tugra customer portal allows to re-issue certificates for already validated domain validation certificates

The vulnerabilities I discovered allowed me to view the password reset emails sent to any customer. As a result, I could log into any customer account, via resetting their password. There is evidence for this in the screenshots of my post. Based on what you have said, this could have compromised the certificate of any E-Tugra customer.

The penetration test and incident report are fine. I am only concerned that E-Tugra does not understand the impact of a severe issue in its own systems. It is important that we are clear that this incident could have impacted E-Tugra's issuance and CA operations. The following statements from E-Tugra appear to be simply false:

We confirm that the affected system is separate from the CA system.
It doesn’t have any direct operational connection with the CA system

Based on the above, it is impossible for the affected system to be completely separated from the CA system. The customer portal cannot be separate from the CA system but also the main way for users to request and obtain certificates.

We checked in our logging system and you attempted multiple times to place an order for the certificate through the portal but could not get the certificate.

I was unable to purchase a certificate in my own account because your website did not accept American credit cards. This is not relevant to the vulnerabilities.

Hi Ian,

We understand the concerns raised in the incident report and potential impact on certificate issuance and CA operations. We performed several steps to address the issue, including remediation measures and making improvements to our systems / processes to ensure the security and integrity of our operations. We executed a thorough investigation of the incident and have found no evidence that the vulnerability was exploited by any malicious actors. We checked all the logs and each of our issued certificates to find any negative effect on our system. We have taken preventive actions and supportive measures to avoid such vulnerabilities in future.

We understand the importance of maintaining the security and integrity of our customers' certificates, and we assure that we are taking care of best security principles in place to ensure the protection of our customers' data and assets. Thanks for your critical input as such feedback is always helpful in strengthening security further.

Hi Ahmed,
Where are we on the remediation measures related to this matter? What action items are still being worked on?
Thanks,
Ben

Flags: needinfo?(ahmed)
Whiteboard: [ca-compliance] Next update 2023-01-06 → [ca-compliance] [uncategorized]

Hi Ben,

We have taken the following remediation measures as part of this incident:

Monitoring processes were enhanced in operations
Internal Tests are enhanced and performed regularly for both CA & Non-CA systems.
Organized Security Training sessions for CA staff
Revised applications acceptance procedure and tests

The only action item which is in-progress is Penetration Testing. We were in discussion with several third party penetration testing companies. We have selected the better one and shared the comprehensive scope with them. Penetration testing company has already started the work. We will keep posting about the updated progress for it.

Ahmed, Please add status updates every week from now until this is fully resolved.

Hi Kathleen,

Apology for not posting updates frequently as we were aggressively working with penetration company considering the importance of this incident. Of-course I will be posting updates as you suggested.

Penetration company has executed the major part of the scope with some minor items in-progress. We got some minor findings reported, fixed them and shared the remediation plan with them. We are expecting they will be closing the scope this week and proceed with the remediation/validation of the fixed points.

In the reports there are no findings surrounding this incident and we are hopeful that results will be very much positive here after the completion of this exercise.

We will share the findings and related reports to auditors during the audit. It would be great if you guide us which points we can share here for the community & google group discussion to close this incident.

Hi,

I am writing to inform you of the results of the penetration testing we conducted on our systems and provide an update on the actions we have taken in response. We applied several security controls in strengthening both internet and intranet systems and penetration testing were performed after we were done with our updates.

Penetration testing company is almost done with the exercise and a couple of areas are left for the intranet tasks. They are also starting the remediation testing for the provided fixes by the end of next week.

We aim that the whole exercise will be completed in this month. We will keep posted updates. Thanks.

Hi Ahmed,
Please make sure that there is some type of executive summary or publicly available report generated by the pentesting company that provides a high-level, non-security-sensitive explanation of what has been accomplished. We need something that can be uploaded here as an attachment.
Thanks,
Ben

Hi,
An update for the pen test process:

  • First phase (covering the full scope) has been completed by pen test company
  • Fixes for some minor findings are also delivered to pen test company
  • Pen test company will start the validation phase next week
    The pen testing company will prepare an executive summary report independently for publishing to the community once they are done with the validation phase.
    Thanks,
    Ahmed

I noticed that according to the timeline, the penetration test started 131 days ago (Nov 18, 2022). A typical penetration test usually lasts a matter of weeks, not nearly half a year. It is difficult to comprehend what a penetration testing firm would be doing for such an extended duration, and I sincerely hope the executive summary of the engagement carries enough detail to clarify this.

One plausible explanation for the long timeline could be an interim remediation and retesting process. If this is the case, it would be valuable for stakeholders to understand the type of fortification that was performed during this period. Additionally, this raises the question of why the annual penetration testing missed these issues in the first place. It would be beneficial for the community if an explanation could be provided on this front, along with a discussion of the changes being implemented to prevent such lapses from happening again in the future.

As we all share a common interest in maintaining a secure and robust ecosystem, it is crucial to address these concerns and ensure transparency in the process. I look forward to hearing more about the specifics of the engagement, the reasons for the extended timeline, and the steps being taken to enhance the overall security posture of the organization.

You need to log in before you can comment on or make changes to this bug.