Closed Bug 1832570 Opened 2 years ago Closed 1 year ago

SSL.com: subCA/Reseller Issues

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cclements, Assigned: secauditor)

References

Details

(Whiteboard: [ca-compliance] [policy-failure])

This report is related to Bug 1801345. For reference, the Chrome Root Program policy states:

Section: Minimum Requirements for CAs

“CA owners with self-signed root CA certificates included in the Chrome Root Store must satisfy the requirements defined in this policy, including taking responsibility for ensuring the continued compliance of all corresponding subordinate CAs and delegated third parties participating in the Public Key Infrastructure (PKI).”

Section: 7. Responding to Incidents

“The failure of a CA owner to meet the commitments of this policy is considered an incident, as is any other situation that may impact the CA’s integrity, trustworthiness, or compatibility.”

The progression of Bug 1801345 has resulted in questions surrounding the process by which SSL.com ensures initial and continued compliance of “managed SubCAs” (possibly referred to as “branded CAs” in Section 1.3.1.2 of the SSL.com CP/CPS). As the corresponding CA owner included in the Chrome Root Store, SSL.com is ultimately responsible for ensuring continued compliance of all corresponding subordinate CAs, like those issued to e-Tugra.

The following questions are intended to better understand how SSL.com ensures continued compliance of all corresponding subordinate CAs.

  1. Can you explain the SSL.com and e-Tugra relationship in the context of the SSL.com CP/CPS? Where in the CP/CPS can we see this relationship structured and defined?

  2. Which other CAs operated under the SSL.com roots included in the Chrome Root Store share this same categorization?

  3. Can you specify under which circumstances and criteria SSL.com delegates elements of the certificate issuance and management processes to the category of CAs identified in Question 1 & 2 (directly above) (e.g., e-Tugra)? This should include the considerations that go into allowing external systems (like the e-Tugra “customer portal”) to be integrated into or influence certificate issuance and management workflows.

  4. Can you specify the mechanics of how that delegation occurs?

  5. Can you describe how SSL.com initially evaluates the underlying security related to these CA customer organization systems capable of influencing certificate issuance or management? Examples of these types of systems include what is referred to as the “e-Tugra panel” and “customer portal” in Bug 1801345.

  6. Can you describe how SSL.com continuously evaluates the underlying security related to these CA customer organization systems capable of influencing certificate issuance or management after performing the initial assessment described in Question 5 (directly above)?

  7. Can you describe change control reporting requirements imposed upon these CA customer organizations related to planned and/or actual changes to systems capable of influencing certificate issuance or management processes?

  8. Can you describe why SSL.com did not detect the system security deficiencies described by Ian Carroll and are further outlined in Bug 1801345?

  9. Can you describe, in detail, technical, policy, and procedural controls or improvements SSL.com has or is planning to make to prevent future instances of the system security deficiencies observed in Bug 1801345? This description should be accompanied by a time-bound commitment of next steps.

  10. Comment 9 correlates dNSNames represented in the administrative panel accessed by Ian Carroll to certificates ultimately issued by SSL.com. We also observed numerous issues related to e-Tugra’s incident response communications.

    a. Can you describe what SSL.com’s role was in responding to Bug 1801345?

    b. Can you describe SSL.com’s responsibility in responding to the concerns addressed in Bug 1801345, given the certificates described in Comment 9 were only trusted in Chrome via transitive trust to SSL.com?

Flags: needinfo?(support)
Blocks: 1799533

(In reply to Chris Clements from comment #0)

This report is related to Bug 1801345. For reference, the Chrome Root Program [policy]

The progression of Bug 1801345 has resulted in questions surrounding the process by which SSL.com ensures initial and continued compliance of “managed SubCAs” (possibly referred to as “branded CAs” in Section 1.3.1.2 of the SSL.com CP/CPS). As the corresponding CA owner included in the Chrome Root Store, SSL.com is ultimately responsible for ensuring continued compliance of all corresponding subordinate CAs, like those issued to e-Tugra.

The following questions are intended to better understand how SSL.com ensures continued compliance of all corresponding subordinate CAs.

  1. Can you explain the SSL.com and e-Tugra relationship in the context of the SSL.com CP/CPS? Where in the CP/CPS can we see this relationship structured and defined?

E-Tugra is not a cross-signing partner. It is strictly a reseller customer to which SSL.com provides hosted SubCA services under an agreement. This is a standard service for which SSL.com issued a SubCA where the subjectDN information matches E-Tugra and the SubCA is explicitly hosted in our systems. SSL.com controls all key materials and aspects of validation and issuance related to this SubCA. There is no external direct control of these SubCAs, nor are they hosted externally from our systems. No other relationship between SSL.com and E-Tugra exists.

This type of service is covered by the reference to “branded” SubCAs in CP/CPS section 1.3.1.2, where the term “branded” SubCA refers to SubCAs which are issued to the name of a reseller, all of which are SubCAs operated internally by SSL.com under the same audited controls we operate all our SubCAs.

  1. Which other CAs operated under the SSL.com roots included in the Chrome Root Store share this same categorization?

All our branded SubCAs are disclosed at the SSL.com public repository under the header “SSL.com Partners”. They are the ones that do not include the name “SSL.com” in the subject:commonName attribute.

  1. Can you specify under which circumstances and criteria SSL.com delegates elements of the certificate issuance and management processes to the category of CAs identified in Question 1 & 2 (directly above) (e.g., e-Tugra)? This should include the considerations that go into allowing external systems (like the e-Tugra “customer portal”) to be integrated into or influence certificate issuance and management workflows.

The customer (reseller) requests their name to be included in the Issuing CA Certificate subject and they sign an appropriate agreement. Resellers are only allowed to submit certificate orders and revocation requests for those certificates; however, the domain validation and the verification of documents is performed by SSL.com.

In particular, as part of submitting a certificate order, the customer (reseller) has no capabilities to delegate or influence the certificate validation workflows, outside of the submission of artifacts on behalf of the subscriber or accessing and providing the appropriate DNS/file change requests to the applicant for domain validation methods 3.2.2.4.7, 3.2.2.4.18. The entire validation/verification including the domain validation/verification is performed by SSL.com and is not delegated to the reseller at any time.

  1. Can you specify the mechanics of how that delegation occurs?

The reseller either uses the RA portal manually or the RA API the same way any customer does. The difference is that the reseller's account is configured to use the branded CAs for certificate issuance instead of the "SSL.com” ones.

  1. Can you describe how SSL.com initially evaluates the underlying security related to these CA customer organization systems capable of influencing certificate issuance or management? Examples of these types of systems include what is referred to as the “e-Tugra panel” and “customer portal” in Bug 1801345.

As described above, SSL.com maintains full control of all SubCAs and does not delegate any Domain Validation tasks to third parties. Resellers with branded SubCAs, such as E-Tugra, have essentially the same type of lifecycle API access as "ordinary" non-SubCA customers and resell branded certificates using their own web portals.

SSL.com does not manage or monitor reseller portals or tools customers use. We consider this to be out of the scope of a CA’s obligations, at least with the current industry policies.

As part of our due diligence, we took this opportunity to revisit the CA – reseller relationship in the industry. We investigated past discussions in m.d.s.p. and located two relevant cases [1], [2], which we intend to use as guidance to promote adoption of good security practices by resellers.

  1. Can you describe how SSL.com continuously evaluates the underlying security related to these CA customer organization systems capable of influencing certificate issuance or management after performing the initial assessment described in Question 5 (directly above)?

Please refer to the response to question 5.

  1. Can you describe change control reporting requirements imposed upon these CA customer organizations related to planned and/or actual changes to systems capable of influencing certificate issuance or management processes?

Please refer to the response to question 5.

  1. Can you describe why SSL.com did not detect the system security deficiencies described by Ian Carroll and are further outlined in Bug 1801345?

Please refer to the response to question 5.

  1. Can you describe, in detail, technical, policy, and procedural controls or improvements SSL.com has or is planning to make to prevent future instances of the system security deficiencies observed in Bug 1801345? This description should be accompanied by a time-bound commitment of next steps.

Please refer to the response to question 5.

  1. Comment 9 correlates dNSNames represented in the administrative panel accessed by Ian Carroll to certificates ultimately issued by SSL.com. We also observed numerous issues related to e-Tugra’s incident response communications.

    a. Can you describe what SSL.com’s role was in responding to Bug 1801345?

We have been monitoring bug 1801345 as part of our public bug monitoring process. Our initial intention was to post a message to clarify that E-Tugra was only acting as a reseller from our point of view, and the issues that affected E-Tugra's systems did not impact on our CA operations. However, we considered that E-Tugra's responses (comment #35, comment #37) made it abundantly clear that their web portal was in no way associated with Domain Validation activities performed by SSL.com.

We were independently contacted by other Root Store Managers for additional clarifications, and we replied with the same information mentioned in this post.

b. Can you describe SSL.com’s responsibility in responding to the concerns addressed in [Bug 1801345](https://bugzilla.mozilla.org/show_bug.cgi?id=1801345), given the certificates described in [Comment 9](https://bugzilla.mozilla.org/show_bug.cgi?id=1801345#c9) were only trusted in Chrome via transitive trust to SSL.com?

E-Tugra orders certificates from SSL.com as a reseller on behalf of their customers. We repeat that SSL.com controls all the validation aspects for the issuance of these certificates without the reseller being able to influence these processes.

In summary, the SubCAs with E-Tugra's name in the subjectDN are no different than any other of SSL.com’s SubCAs. E-Tugra was acting solely as a reseller of SSL.com certificates and has had the same control as any standard customer who signs up in our RA Portal and uses it to request certificates and (re-)sell them to other end customers.

Applicants or resellers (on behalf of the Applicants) can place orders and submit evidence via API or the web UI, and SSL.com performs the required validation of these order requests. If Emails are used for Domain Validation purposes, these emails are sent directly from SSL.com’s systems to the Applicant’s designated validation email address.

We hope the above answers are clear and address the concerns about the business relationship between SSL.com and E-Tugra.

Flags: needinfo?(support)

Hi Thomas,

Regarding the previous questions #1 and #2, thank you for clarifying the relationship between SSL.com and e-Tugra as that of a reseller (as this was not obvious to us in bug 1801345, nor is it disclosed in centralized sources like CCADB) and for identifying all SSL.com current resellers.

It is strictly a reseller customer to which SSL.com provides hosted SubCA services under an agreement.

To help us better understand this specific reseller relationship:

  1. Can you share if and how SSL.com evaluates resellers before entering a contractual relationship? For example, does SSL.com consider the reseller's business practices, technical capabilities, and adherence to industry standards, like those system security controls described in the Baseline Requirements?
  2. Can you share if and how the agreement or Reseller Terms accounts for establishing minimum expectations related to the security of reseller systems involved in certificate issuance and management (e.g., systems that submit requests to the CA on behalf of the subscriber, or initiate revocation requests)?
  3. Does anything in the agreement or Reseller Terms account for instances like this e-Tugra incident where reseller privileged systems are exploited and able to further compromise accounts on reseller customer portals? For example, is a reseller required to provide evidence of security incident response procedures?

As part of our due diligence, we took this opportunity to revisit the CA – reseller relationship in the industry. We investigated past discussions in m.d.s.p. and located two relevant cases [1], [2], which we intend to use as guidance to promote adoption of good security practices by resellers.

  1. Can you elaborate on the specific guidance that SSL.com plans to offer to its resellers because of this e-Tugra incident?

We have been monitoring bug 1801345 as part of our public bug monitoring process. Our initial intention was to post a message to clarify that E-Tugra was only acting as a reseller from our point of view, and the issues that affected E-Tugra's systems did not impact on our CA operations. However, we considered that E-Tugra's responses (comment #35, comment #37) made it abundantly clear that their web portal was in no way associated with Domain Validation activities performed by SSL.com.

  • The e-Tugra response to bug 1801345 lacks clarity, evidenced by not only our continued questions, but several questions/comments from the public:

    • Several questions were not answered by e-Tugra
    • Several contradicting statements were made by e-Tugra
    • Several statements made by e-Tugra were refuted
    • e-Tugra failed to meet several commitments made in response to the bug
    • The output of the penetration test (i.e., reports) attached by e-Tugra do not offer any detail addressing concerns raised in the incident report, nor do they corroborate statements made by e-Tugra related to architecture and connectivity

  • Comment #35 and #37 occurred roughly 5 months after the incident report was created and over 4 months after SSL.com was identified as the pre-certificate issuer in comment #9. A timely response acknowledging awareness and monitoring of the incident with clarity and confirmation from SSL.com as the CA owner would have been welcomed and appreciated (e.g., similar to this recent reseller event). We feel that SSL.com, as the CA owner, has a larger role in the timely and transparent communication to everyone in evaluating the nature of the incident. We plan to explore these expectations such that we can more clearly present this opinion in the future, possibly resulting in an update to our program policy or the Baseline Requirements. We do acknowledge the single clarification provided by SSL.com in comment #34.

In summary, the SubCAs with E-Tugra's name in the subjectDN are no different than any other of SSL.com’s SubCAs. E-Tugra was acting solely as a reseller of SSL.com certificates and has had the same control as any standard customer who signs up in our RA Portal and uses it to request certificates and (re-)sell them to other end customers.

  1. Can you elaborate on how the e-Tugra branded SubCAs would respond to a certificate request originating from the e-Tugra customer portal (described by Ian), where an account takeover has occurred (enumerated by Ian) and previous domain validation already took place for the requested domain within the previous 397 days? e-Tugra described the “e-Tugra customer portal allowing re-issuance for already validated domains” in comment #12.

  2. Can you elaborate on how the e-Tugra branded SubCAs would respond to a revocation request originating from the e-Tugra customer portal (described by Ian), where an account takeover has occurred (enumerated by Ian)?

Hello Chris,

We are analyzing the questions and plan to respond by EOD on Friday.

Thank you,

Tom
SSL.com

(In reply to Chris Clements from comment #3)

Hi Thomas,

Hi Chris,

Thank you for your patience; we wanted to provide accurate and complete information to each question below. If additional clarity is needed, we are happy to oblige.

Before we dive into the questions, we would like to elaborate more on two areas to provide some context:

  1. Resellers and their role in the ecosystem
  2. Options SSL.com offers to large customers (and which can be leveraged by resellers, hosting providers, or other customers who need large volumes of certificates).

This information will allow the interested reader to have a better understanding of our answers and promote discussion in the community regarding a CA’s responsibilities over resellers.

Resellers and their role in the ecosystem

Per our understanding, resellers are natural or legal persons (third parties) that, while re-packaging certificates and re-selling them (via their own channels or customer-facing systems), assist subscribers with the certificate issuance and management processes that some subscribers find cumbersome. When subscribers choose to use a reseller instead of directly interacting with the original issuer (CA), they purposely and willfully consent to the delegation of some authority to the reseller. Subscribers may receive additional services from a reseller, like 24/7 support, tech visits on site, and other added-value services that the CA may not provide.

We would like to describe an illustrative example of a common “reseller use case”, hosting providers. A significant portion of website/domain owners don’t have the IT skills or capacity, or simply don’t want to be involved in setting up and managing the web server that hosts their websites, including the issuance and installation of the TLS certificate. For this reason, they partially or entirely delegate the domain validation, certificate issuance and installation process to a hosting provider of their choice. In the typical scenario, the hosting provider creates an account at the CA’s web portal, orders the TLS certificate, completes the DV challenge (either on their own, if authorized, or via the assistance of the website/domain owner), receives and installs the certificate. The hosting provider may do this manually or use automation.

The above demonstrates that the CA has no way to distinguish whether the above actions were performed by the same or different entities, i.e. as far as the CA is concerned, since these two entities (reseller, subscriber) are represented by the same user account, they are one and the same. This is a common practice which was never questioned in the past by the industry. Discussions such as this one demonstrate that this “silent” delegation of parts of the certificate lifecycle to resellers, even critical ones such as the management of subscriber private keys, is an acknowledged and accepted practice. Recent discussions (references: 1, 2, 3, 4, 5) in the Validation Subcommittee of the Server Certificate WG are additional indications that this delegation is a known and accepted practice.

SSL.com options for large customers

SSL.com offers the following options to larger or high-volume customers:

  1. Anyone who is interested in large volumes of certificates may participate in the Reseller and Volume Purchasing (RVP) Program and benefit from volume discounts. As the page says, these may be customers who just need large volumes of certificates for in-house use, or customers who want to resell certificates along with other services, such as hosting providers. They sign-up to the RVP Program on their own, by creating an account via a dedicated registration page and accepting the standard Subscriber Agreement, without specifying whether they will resell the certificates or use them in-house. The primary incentive in joining the RVP Program are the financial benefits (volume discounts).

  2. Anyone who is interested in promoting their name as the issuer of certificates, without having to invest in PKI infrastructure and staff, may request a branded subCA as described in the Custom-Branded Issuing CA page. To receive a branded subCA, they sign a “SubCA and Reseller Agreement”, which is partially customized to their needs. As the page says, custom-branded issuing CA certificates are signed and hosted by SSL.com, who controls all key materials and aspects of validation (domain and identity) and issuance, and goes through “regular reviews, testing and rigorous annual external audits to provide peace of mind”, i.e., they are exclusively internally-operated. Most often these are companies who would like to resell certificates branded to their name. Note that these are the customers that are disclosed under section “SSL.com Partners” of our Online Repository.

All SSL.com customers (plain user accounts, large customer/reseller accounts such as hosting providers, and branded subCA owners) may make use of the same systems (SSL.com portal and API) with the same access rights for their needs. If a customer, other than branded subCA owner, re-packages SSL.com products and services and (re-)sells them via its own channels or customer-facing systems, this information would not be available to us. To the best of our knowledge, this applies to all CAs and retail/commercial sales in general.

All the above indicate that SSL.com makes no distinction between large customers and resellers of any kind, except for branded subCA owners.

Regarding the previous questions #1 and #2, thank you for clarifying the relationship between SSL.com and e-Tugra as that of a reseller (as this was not obvious to us in bug 1801345, nor is it disclosed in centralized sources like CCADB) and for identifying all SSL.com current resellers.

E-Tugra’s name and the branded subCAs information have been listed under the “SSL.com Partners” section of our Online Repository since the issuance of the branded subCA certificates by SSL.com. This is a standard SSL.com practice that we apply for transparency reasons. Similarly, all branded subCA certificates are included in our audit reports and in CCADB, where they are marked as having the same audit report as the parent CA certificates, indicating that these are internally operated subCAs. We are not aware of any other flags in CCADB to disclose this information in a better way; should they exist, we would be happy to use them. Also, looking back to our Comment #34 in bug 1801345, it becomes clear that we focused solely on addressing the specific issue at hand, namely to clarify that this has been strictly an internally-operated subCA and that there has been no cross-certification between SSL.com and E-Tugra.

It is strictly a reseller customer to which SSL.com provides hosted SubCA services under an agreement.

To help us better understand this specific reseller relationship:

  1. Can you share if and how SSL.com evaluates resellers before entering a contractual relationship? For example, does SSL.com consider the reseller's business practices, technical capabilities, and adherence to industry standards, like those system security controls described in the Baseline Requirements?
  2. Can you share if and how the agreement or Reseller Terms accounts for establishing minimum expectations related to the security of reseller systems involved in certificate issuance and management (e.g., systems that submit requests to the CA on behalf of the subscriber, or initiate revocation requests)?
  3. Does anything in the agreement or Reseller Terms account for instances like this e-Tugra incident where reseller privileged systems are exploited and able to further compromise accounts on reseller customer portals? For example, is a reseller required to provide evidence of security incident response procedures?

After providing our understanding of the current resellers model and presenting the options we offer to large customers, the answers to questions 1-3 are as follows:

  1. Members of the RVP Program (the offering described in bullet 1 of section “SSL.com options for large customers” above), who create and use their account without the intervention of SSL.com personnel, are considered subscribers, and thus are not subject to any evaluations or vetting. For branded SubCA customers/partners (the offering described in bullet 2 of section “SSL.com options for large customers” above), after vetting the organization, we engage into discussions on the use of the branded subCAs, the exact responsibilities of each party, the modalities which are available for the partner to interact with our systems (SSL.com portal and API) and the proper ways to do so. We do not perform any security audits on their portals or detail specific system security controls such as those described in the BRs for them to follow.

  2. All Subscribers, including resellers, are bound to our CP/CPS and main Subscriber Agreement located in the SSL.com Repository. These include terms that require subscribers to establish a secure environment to protect private keys, though without more specific practices such as those described in the CA/B Forum Network and Certificate Systems Security Requirements. In particular, paragraph 3.3 of the Subscriber Agreement requires them to “take all reasonable measures to protect their Certificate Services account, their Private Key, and any associated activation data or device (such as a password, pass phrase, or token) from unauthorized use and disclosure”. Branded SubCA subscribers sign a “SubCA and Reseller Agreement” that includes more restrictions and warranties, such as requirements for the proper training of the subCA’s personnel, protection of any confidential information and the obligation of the subCA to retain records for audit/inspection purposes.

  3. All subscribers, including resellers, must report security incidents that might have an adverse effect on certificates issued by SSL.com. For example, paragraphs 3.6 and 3.7 of our Subscriber Agreement require subscribers to notify SSL.com if they discover or suspect that their Certificate Services account has been compromised, and that they cooperate with SSL.com concerning any compromise. We do not require subscribers to provide evidence of their specific security incident response procedures.

As part of our due diligence, we took this opportunity to revisit the CA – reseller relationship in the industry. We investigated past discussions in m.d.s.p. and located two relevant cases [1], [2], which we intend to use as guidance to promote adoption of good security practices by resellers.

  1. Can you elaborate on the specific guidance that SSL.com plans to offer to its resellers because of this e-Tugra incident?

We are planning to create a public guide available for subscribers/resellers that will cover, at a minimum, the following topics:

  • How to secure and maintain a secure website (guidance for proper TLS server configuration, operating system hardening, common website vulnerabilities)
  • Secure storage of private keys. Discourage subscriber/resellers from storing private keys into their management portals, highlighting the expectations of the BRs. In the reseller case, if they have explicit authorization by a subscriber to store a private key, this will need to be done in an encrypted manner to reduce the risk of private key exposure to a potential attacker
  • Use CA-vetted tools for generating keys and CSRs
  • Promote the use of multi-factor authentication for actions that trigger any CA API interaction (certificate re-key, renewal, revocation).
  • Promote implementation of the CA/B Forum Network and Certificate Systems Security Requirements with emphasis on quarterly vulnerability scans and annual penetration testing.

Additionally, we plan to revisit the current agreement documents to differentiate between subscribers who wish to use certificates in-house and those who plan to resell them. Some of the improvements we are considering are:

  • Enforce Reseller Agreement that allows re-selling, issuance, revocation and management of certificates on behalf of other entities, under additional provisions, such as:
    • Requirement to follow secure practices to protect subscribers, with reference to the public guide described above.
    • Requirement to report security incidents that may compromise the security of subscribers so the CA may assess the impact and work with the reseller to mitigate the risks.
  • Update the standard Subscriber Agreement to discourage subscribers from re-selling certificates or managing them on behalf of another entity, thus redirecting them to the above Reseller Agreement.

We have been monitoring bug 1801345 as part of our public bug monitoring process. Our initial intention was to post a message to clarify that E-Tugra was only acting as a reseller from our point of view, and the issues that affected E-Tugra's systems did not impact on our CA operations. However, we considered that E-Tugra's responses (comment #35, comment #37) made it abundantly clear that their web portal was in no way associated with Domain Validation activities performed by SSL.com.

  • The e-Tugra response to bug 1801345 lacks clarity, evidenced by not only our continued questions, but several questions/comments from the public:

    • Several questions were not answered by e-Tugra
    • Several contradicting statements were made by e-Tugra
    • Several statements made by e-Tugra were refuted
    • e-Tugra failed to meet several commitments made in response to the bug
    • The output of the penetration test (i.e., reports) attached by e-Tugra do not offer any detail addressing concerns raised in the incident report, nor do they corroborate statements made by e-Tugra related to architecture and connectivity

  • Comment #35 and #37 occurred roughly 5 months after the incident report was created and over 4 months after SSL.com was identified as the pre-certificate issuer in comment #9. A timely response acknowledging awareness and monitoring of the incident with clarity and confirmation from SSL.com as the CA owner would have been welcomed and appreciated (e.g., similar to this recent reseller event). We feel that SSL.com, as the CA owner, has a larger role in the timely and transparent communication to everyone in evaluating the nature of the incident. We plan to explore these expectations such that we can more clearly present this opinion in the future, possibly resulting in an update to our program policy or the Baseline Requirements. We do acknowledge the single clarification provided by SSL.com in comment #34.

We understand the resulting sentiment and we will make sure to engage in such discussions earlier. From our perspective, the early discussions focused on E-Tugra's portal, the security evaluations/pen-test and whether it could have been used to directly cause the issuance of any arbitrary certificate bypassing Domain Validation and other controls required by the BRs.

As the reseller-CA topic had been covered in past discussions, the need to intervene to the bug became clear when we realized that there was a possible misinterpretation of our relationship with E-Tugra. Particularly, comment #33 suggested that there is a cross-signing relationship between the two entities, that E-Tugra had control of keys associated with a CA, and that they could perform validation and cause issuance to any domain name. Our response/clarification to that possible misinterpretation was posted on the same day.

In summary, the SubCAs with E-Tugra's name in the subjectDN are no different than any other of SSL.com’s SubCAs. E-Tugra was acting solely as a reseller of SSL.com certificates and has had the same control as any standard customer who signs up in our RA Portal and uses it to request certificates and (re-)sell them to other end customers.

  1. Can you elaborate on how the e-Tugra branded SubCAs would respond to a certificate request originating from the e-Tugra customer portal (described by Ian), where an account takeover has occurred (enumerated by Ian) and previous domain validation already took place for the requested domain within the previous 397 days? e-Tugra described the “e-Tugra customer portal allowing re-issuance for already validated domains” in comment #12.

  2. Can you elaborate on how the e-Tugra branded SubCAs would respond to a revocation request originating from the e-Tugra customer portal (described by Ian), where an account takeover has occurred (enumerated by Ian)?

It appears there may be some confusion regarding E-Tugra’s own customer portal and the level of access it has (or E-Tugra personnel have) in the SSL.com interfaces. Resellers such as E-Tugra have no additional access to our systems than standard subscribers. A user account in the SSL.com portal is completely separated from a user account in E-Tugra’s own customer portal. E-Tugra is using their own user account credentials to access the SSL.com interfaces and process orders on behalf of their end customers. E-Tugra is responsible for keeping the credentials of their account secure, similarly to any other subscriber.

In general, if a subscriber’s credentials have been compromised, and her/his Certificate Services account (i.e. a user account in the SSL.com portal) has been taken over, the attacker would be able to access the SSL.com customer portal account (and API) and process new or existing orders to request certificate issuances or revocations. Of course, for any issuance, the attacker would have to be able to satisfy DCV, with new or existing non-expired evidence.

However, in the case of reseller user accounts (such as E-Tugra’s), additional restrictions apply to prevent DCV information from being reused in different or new certificate orders. These restrictions are activated once SSL.com determines an account to be acting as a reseller, either by self-declaration by the user or because there is a branded subCA agreement for the user account in question. Therefore, such a compromised account would only be allowed to re-process a previously validated certificate within the same order. New certificate orders would require domain validation to be explicitly performed even for previously validated domains.

Please excuse our extended analysis in this post, but we would like to be as transparent as possible, hoping to assist all stakeholders to get a better understanding of the different use cases. We welcome other CA representatives to state their opinion if they implement additional restrictions or have different approaches to these use cases. Perhaps a CCADB communication (from Chrome or Mozilla) to all participating CAs would be able to collect more feedback that can be used to better assess these use cases.

We remain at your disposal for any additional clarifications.

Hi Thomas,

Thank you for the candid and very detailed response! The extended analysis (especially the additional background information related to resellers) is very much appreciated and helps improve perspective and understanding.

We do have a few more questions related to comments in your response:

  1. All Subscribers, including resellers, are bound to our CP/CPS and main Subscriber Agreement located in the SSL.com Repository. These include terms that require subscribers to establish a secure environment to protect private keys, though without more specific practices such as those described in the CA/B Forum Network and Certificate Systems Security Requirements. In particular, paragraph 3.3 of the Subscriber Agreement requires them to “take all reasonable measures to protect their Certificate Services account, their Private Key, and any associated activation data or device (such as a password, pass phrase, or token) from unauthorized use and disclosure”.
  1. Can you please describe what a “Certificate Services account” is in more detail (to include a description of this account’s credentials)? We can generally interpret what it means, but more specificity would like to improve our understanding.
  2. In this case, we assume that a) e-Tugra’s “Certificate Services account” was not directly compromised (there’s no evidence to suggest it was), and b) that it could not have been compromised through any of the account takeover attacks described by Ian (e.g., observing emails sent from SSL.com to e-Tugra containing a password reset code). However, it would appear that an e-Tugra system with the ability to leverage the privileges of e-Tugra’s Certificate Services account (possibly by way of an automated API call triggered after satisfying pre-requisite activities taking place in the e-Tugra “Customer Portal” that was compromised) could have facilitated abuse/misuse for existing e-Tugra customers. Is that understanding correct?
  3. Related to the leaf certificates identified in this incident, for clarity, who is considered the subscriber (e-Tugra as the reseller, or their customers)?
  4. Do you have any indication that e-Tugra was generating private keys on behalf of their customers? Understandably, this would not be made evident based on contents of the CSRs provided to SSL.com.

Branded SubCA subscribers sign a “SubCA and Reseller Agreement” that includes more restrictions and warranties, such as requirements for the proper training of the subCA’s personnel, protection of any confidential information and the obligation of the subCA to retain records for audit/inspection purposes.

  1. Is the “SubCA and Reseller Agreement” publicly available?
  2. Did the system security and data protection failures demonstrated by e-Tugra through this incident violate any portion of that agreement?
  1. All subscribers, including resellers, must report security incidents that might have an adverse effect on certificates issued by SSL.com. For example, paragraphs 3.6 and 3.7 of our Subscriber Agreement require subscribers to notify SSL.com if they discover or suspect that their Certificate Services account has been compromised, and that they cooperate with SSL.com concerning any compromise. We do not require subscribers to provide evidence of their specific security incident response procedures.
  1. Did e-Tugra share any notification to SSL.com as a result of the vulnerabilities demonstrated by Ian (possibly related to the “... or suspected compromise of your Private Key (s) or Certificate Services, …” language in 3.7 of the Subscriber Agreement)? Based on responses near the end of comment #5, it appears there are still potential scenarios where domain validation reuse could have been performed, or that an unauthorized third party with access to the e-Tugra customer portal could have triggered revocation of a certificate for which they were unauthorized to act upon.

We are planning to create a public guide available for subscribers/resellers that will cover, at a minimum, the following topics:

  • The minimum topics suggested and plan to revisit current agreements seems beneficial. Thank you for outlining these enhancements.

However, in the case of reseller user accounts (such as E-Tugra’s), additional restrictions apply to prevent DCV information from being reused in different or new certificate orders. These restrictions are activated once SSL.com determines an account to be acting as a reseller, either by self-declaration by the user or because there is a branded subCA agreement for the user account in question. Therefore, such a compromised account would only be allowed to re-process a previously validated certificate within the same order. New certificate orders would require domain validation to be explicitly performed even for previously validated domains.

  1. It’s not clear to us what constitutes an “order” in this branded SubCA scenario or when one starts and ends. Can you please elaborate? For example, would it have been possible to place a “multi-year” order that would have allowed domain validation reuse on the 395th day of the original certificate’s validity period?

Hello Chris,

This is to inform you that a full response will be provided by the end of the week, but we did get confirmation directly from e-Tugra on question 4 and we decided to post this sooner than later.

So, specifically for question 4, we have no indication that e-Tugra was generating private keys on behalf of their subscribers. Furthermore, we received a formal statement that e-Tugra has never generated or stored private keys on behalf of subscribers of SSL.com certificates within their portal.

(In reply to Chris Clements from comment #6)

Hi Thomas,

Thank you for the candid and very detailed response! The extended analysis (especially the additional background information related to resellers) is very much appreciated and helps improve perspective and understanding.

Hello Chris,

I am happy to help and hope that these answers will continue to bring the subject to light.

We do have a few more questions related to comments in your response:

  1. All Subscribers, including resellers, are bound to our CP/CPS and main Subscriber Agreement located in the SSL.com Repository. These include terms that require subscribers to establish a secure environment to protect private keys, though without more specific practices such as those described in the CA/B Forum Network and Certificate Systems Security Requirements. In particular, paragraph 3.3 of the Subscriber Agreement requires them to “take all reasonable measures to protect their Certificate Services account, their Private Key, and any associated activation data or device (such as a password, pass phrase, or token) from unauthorized use and disclosure”.
  1. Can you please describe what a “Certificate Services account” is in more detail (to include a description of this account’s credentials)? We can generally interpret what it means, but more specificity would like to improve our understanding.

A Certificate Services Account (CSA) is represented by the unique username/email and password combination used to access the SSL.com certificate portal.

  1. In this case, we assume that a) e-Tugra’s “Certificate Services account” was not directly compromised (there’s no evidence to suggest it was), and b) that it could not have been compromised through any of the account takeover attacks described by Ian (e.g., observing emails sent from SSL.com to e-Tugra containing a password reset code). However, it would appear that an e-Tugra system with the ability to leverage the privileges of e-Tugra’s Certificate Services account (possibly by way of an automated API call triggered after satisfying pre-requisite activities taking place in the e-Tugra “Customer Portal” that was compromised) could have facilitated abuse/misuse for existing e-Tugra customers. Is that understanding correct?

Regarding a) SSL.com shares the same understanding.

Regarding b) password reset codes are sent via email directly from the SSL.com systems to the registered email address associated with the CSA, without the interaction of the E-Tugra portal. To take over the CSA account, it would require other systems to be breached (email account), which are out of scope for SSL.com or any other CA, in terms of monitoring and/or prevention.

Regarding ability of the E-Tugra system to leverage the privileges of E-Tugra’s CSA in the SSL.com portal: It is reasonable to assume that the CSA credentials SSL.com supplied to E-Tugra are used at some point in the process for E-Tugra to authenticate requests against the SSL.com RA API. Based on the information shared by E-Tugra, the various screenshots and the comments posted in https://bugzilla.mozilla.org/show_bug.cgi?id=1801345, our understanding is that these credentials were not accessible by the takeover attack.

  1. Related to the leaf certificates identified in this incident, for clarity, who is considered the subscriber (e-Tugra as the reseller, or their customers)?

The “SubCA and Reseller Agreement” specifies that Sub-CA's end customers of Certificate Products issued by SSL.com are considered Subscribers of SSL.com. It requires Sub-CAs (E-Tugra in this case) to ensure that each Subscriber becomes a party to the SSL.com Subscriber Agreement found at the SSL.com public repository, as it may be updated from time to time during the term of the Agreement, or terms that are at least as protective of SSL.com as those stated in the SSL.com Subscriber Agreement and that have been approved in advance in writing by SSL.com (the “Subscriber Terms”). The Agreement also requires Sub-CAs to enforce the Subscriber Terms to the Subscribers.

Note that we have been monitoring related discussions at the CAB/Forum Server Certificate WG Validation Subcommittee under the topic “Applicant and Applicant Representative” since August 2022, and we plan to remain aligned should there be any development or different expectations on these issues.

  1. Do you have any indication that e-Tugra was generating private keys on behalf of their customers? Understandably, this would not be made evident based on contents of the CSRs provided to SSL.com.

We have no such indication. To the best of our knowledge, E-Tugra was not involved in the generation of private keys on behalf of their customers. Furthermore, upon communication with e-Tugra, we received a formal statement that E-Tugra has never generated and stored private keys on behalf of subscribers for SSL certificates within their portal.

Branded SubCA subscribers sign a “SubCA and Reseller Agreement” that includes more restrictions and warranties, such as requirements for the proper training of the subCA’s personnel, protection of any confidential information and the obligation of the subCA to retain records for audit/inspection purposes.

  1. Is the “SubCA and Reseller Agreement” publicly available?

The “SubCA and Reseller Agreement” is a private document which is based on an internal template, and it is customized per partner to address the particularities of each individual case.

  1. Did the system security and data protection failures demonstrated by e-Tugra through this incident violate any portion of that agreement?

We consider that the question of whether these failures constitute a violation of the “SubCA and Reseller Agreement” is primarily of a legal nature. However, we would like to offer the following comments, based on our understanding and without having reviewed this issue with our legal department.

As we mentioned in our previous post, the severe failures reported in bug #1801345 have been monitored by SSL.com since the beginning of the issue. Our evaluation, based on the information and the screenshots disclosed therein, concluded that these failures did not affect SSL.com systems and operations.

Any personal data related to SSL.com Subscribers, including e-Tugra’s customers, have been securely processed and stored by SSL.com. We have had no indication that the system security failures presented in bug #1801345 had any security impact in SSL.com systems or operations.

Review of the “SubCA and Reseller Agreement” did not reveal any violations related to the protection of personal data of e-Tugra’s customers. Any software that is developed, created, and published independently by E-Tugra falls under the privacy and/or security policies of E-Tugra. Having said that, organizations are obliged to follow the data protection legislation applicable to their jurisdiction.

As stated in Comment #5, we plan to revisit the current agreement documents and add requirements related to the Reseller’s obligation to follow secure practices to protect subscribers and duly report any incidents.

  1. All subscribers, including resellers, must report security incidents that might have an adverse effect on certificates issued by SSL.com. For example, paragraphs 3.6 and 3.7 of our Subscriber Agreement require subscribers to notify SSL.com if they discover or suspect that their Certificate Services account has been compromised, and that they cooperate with SSL.com concerning any compromise. We do not require subscribers to provide evidence of their specific security incident response procedures.
  1. Did e-Tugra share any notification to SSL.com as a result of the vulnerabilities demonstrated by Ian (possibly related to the “... or suspected compromise of your Private Key (s) or Certificate Services, …” language in 3.7 of the Subscriber Agreement)? Based on responses near the end of comment #5, it appears there are still potential scenarios where domain validation reuse could have been performed, or that an unauthorized third party with access to the e-Tugra customer portal could have triggered revocation of a certificate for which they were unauthorized to act upon.

We were notified about the public incident and have been tracking the issue since. E-Tugra did notify us via email raising our attention to the bug, which, as said, we were already monitoring. Although there are possible scenarios where existing certificates could be revoked or reissued (within the same “order”, see below) based on the hosting provider/reseller model we described in our previous post, E-Tugra confirmed that according to their investigation they found no evidence that an unauthorized third party triggered revocation or reissuance of certificates.

However, in the case of reseller user accounts (such as E-Tugra’s), additional restrictions apply to prevent DCV information from being reused in different or new certificate orders. These restrictions are activated once SSL.com determines an account to be acting as a reseller, either by self-declaration by the user or because there is a branded subCA agreement for the user account in question. Therefore, such a compromised account would only be allowed to re-process a previously validated certificate within the same order. New certificate orders would require domain validation to be explicitly performed even for previously validated domains.

  1. It’s not clear to us what constitutes an “order” in this branded SubCA scenario or when one starts and ends. Can you please elaborate? For example, would it have been possible to place a “multi-year” order that would have allowed domain validation reuse on the 395th day of the original certificate’s validity period?

An SSL/TLS certificate order basically is a logical container comprised of one or more CSRs and one or more domains requested by the applicant. Once the domains are validated, a certificate using any of the CSRs assigned to the account can be issued. For a reseller account such as E-Tugra's, the DCV evidence cannot be applied to other certificate orders, and thus requires revalidation of those domains in new orders, even if the orders are concurrently active.

With regards to the maximum validity period of TLS certificates and the maximum re-use period of DCV evidence, SSL.com follows the Baseline Requirements. A multi-year TLS certificate order is served by issuance of multiple single-year certificates, with each (typically) issued some time before the expiration of the previous one. In the case of a non-reseller account, the system allows re-use of any valid non-expired DCV evidence tied to the Subscriber’s account (from the same or other orders). In the case of a reseller account, DCV re-use is possible within the same order, therefore in the scenario described in your question, the reseller would be allowed to complete the first renewal.

Hi Thomas,

We asked multiple questions to improve our understanding, and we appreciate your detailed responses, which helped clarify the impact of 1801345.

We equally appreciate SSL.com’s commitments to:

  1. create public guidance for subscribers/resellers as described in Comment #5
  2. revisit the current agreement documents to strengthen obligations and practices as described in Comment #8

Please let us know at chrome-root-program [at] google [dot] com when these activities are completed. Otherwise, we have no further questions for this report. Again, thank you.

Thank you Chris,

We have already scheduled a meeting next week with the appropriate business units to discuss and decide on a timeline for both of those topics. We will post the timelines to this bug no later than Friday June 23.

For tracking the pending actions, we are fine with either leaving this bug open (preferably without the “CA Certificate Compliance” tag) or creating a new one.

Hi Chris,

We would like to update you on SSL.com’s initiative to create a white paper with lessons learned from this bug and from relevant discussions regarding the model of internally operated, branded SubCA resellers, related security implications and recommended practices.

Since this model is likely used by several other CAs, we believe it is important to leverage the experience of the industry. For this reason, we have identified the top 10 CAs with the most intermediate certificates and, from these, we plan to identify the ones selling branded SubCAs. The intention is to reach out to the other CAs with similar reseller models and ask for their contribution in the preparation of such a white paper. To this end, we are relying on information from the CCADB. Any CA that wishes to contribute may also reach out to secauditor [at] SSL [dot] com.

Our estimated time of completion for the initiatives are as follows:

  1. Creation of guidance for resellers - End of August
  2. Amendment of agreement documents and generation of terms/conditions for resellers and branded SubCA customers – End of year (rough estimation due to the legal implications of the issue)
  3. Collaborative white paper on the lessons learned, security implications and good practices for branded SubCAs – End of October (1 month to reach out to the other CAs and 3 months to collaborate and develop a white paper).

We appreciate any suggestions from the root store owners and the community.

Sincerely,

Tom
SSL.com

Tom, thank you for the update. We appreciate the approach described in your plan, and agree that through collaboration, members of the community can work together to share their experiences and best practices - ultimately amplifying the positive outcome of this effort.

Responding to Comment 10, I suggest we continue to track progress and output in this existing bug, but plan on changing the status to “Resolved” and resolution of “Invalid”. I believe that aligns with past precedent for noting activity that ultimately results in a non-incident.

Thank you, Chris.
We appreciate the change in status and will continue to provide updates of our progress to this bug.
Cheers.

Summary: SSL.com: e-Tugra Security Issues → SSL.com: subCA/Reseller Issues

We would like to update the community on the progress of our 3 commitments. The first of which, a Reseller Guidance article, is in the final stages of review and will be publicly released by the end of this week. Our other two commitments are moving forward and we will work as diligently as possible to meet their original timelines. Thank you for your patience and understanding.

I would like to update the community on our progress. The reseller document was unfortunately delayed, but will be published this week.

This update is to report that the Reseller Guidance article has been published to https://www.ssl.com/guide/certificate-authority-security-best-practices-guide-for-branded-resellers-comprehensive-security-measures/

This concludes initiative #1. The second and third initiatives are still progressing and we will update this bug periodically with status summaries. Thank you.

Whiteboard: [ca-compliance] [uncategorized] → [ca-compliance] [policy-failure]
Whiteboard: [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure] Next update 2023-12-01

We currently have no new news to share. We will provide any updates by 12/15/2023 (2 weeks from today).

Thomas,
Can you provide a final update, and then we can close this out?
Thanks,
Ben

Flags: needinfo?(support)

We thank the community for your patience as we approach completing the outlined tasks and delivering a white paper on “Lessons learned, security implications and good practices” for branded SubCAs, compiled from various sources and our own experiences.

The following tasks have been completed thus far:

  • Analysis of CCADB to identify CAs with the most experience with the branded subCAs model, and extract useful information regarding Intermediate CA Certificates (internally and externally operated)

  • Preparation of a survey to solicit information from those CAs

  • Analysis of results to identify commonalities and differences in the branded subCA model as applied by the CA industry

  • Preparation of an initial draft of good practices at the business level, complementary to the Security Best Practices Guide for Branded Resellers we published in October.

Leveraging the work done for the first draft, an extended revision is underway to include survey information and provide coverage of risks related to other types of resellers (e.g. Value-Added Resellers) and the respective good practices. Factoring in reviews and approvals by the CAs that participated in the survey, we expect to have a finalized document published within February.

Flags: needinfo?(support)
Whiteboard: [ca-compliance] [policy-failure] Next update 2023-12-01 → [ca-compliance] [policy-failure] Next update 2024-02-09

We would like to inform the community of our progress towards action item 3, the whitepaper. This week we have completed the outline of the 2nd draft of the document and we are now in the authoring stage. The process is on track to meet the recent estimates; we will update the bug in two weeks (2024-02-02) with another progress report.

Work on the white paper is progressing. We will provide an update next week.

Whiteboard: [ca-compliance] [policy-failure] Next update 2024-02-09 → [ca-compliance] [policy-failure]

The 3rd initiative, the creation of a whitepaper on the lessons learned, security implications and good practices for branded SubCAs is being finalized, pending clarifications from some CAs who participated in the initial survey. We expect to publish the document before the end of March. We would also like to express our deep regret for omitting any updates to the bug.

No longer blocks: 1799533
See Also: → 1799533

This is an update to inform about our progress with the white paper.

The content of the white paper has been completed from our side.

We will share the document at Google Drive for comments / suggestions from the community. The plan is to provide a two weeks period for feedback. During this time, we will also work on the layout and aesthetics to get the paper ready for publication.

Thank you for your patience and understanding.

We would like to invite the community to review and comment on the Lessons Learned Whitepaper, available at:

https://docs.google.com/document/d/1XdIilN6UE0h2kngD3IbLTVp2LnspR5m7awhgWuklN98/edit?usp=sharing

The document will be available for the rest of the week, at which time we will make any final changes based on community feedback and publish the final edition by Friday, May 3.

Correction to the timeline:

The document will be available until Friday, May 3 for feedback/comments by the community. SSL.com will process the feedback and plan to publish the final edition by Friday, May 10.

I've added my comments and suggestions to the draft. Please review and adjust it accordingly. Thanks.

Flags: needinfo?(support)

This marks the close of the contributions period. Thanks to all who contributed, especially Ben Wilson. SSL.com has accepted all editorial changes and is working on the questions and suggestions to improve and publish the Lessons Learned Whitepaper.

SSL.com has processed the feedback, accepted most recommendations, and drafted responses to the comments that need another round of discussion. We intend to post our responses to the draft document no later than May 14.

Flags: needinfo?(support)

We have responded to all comments on the Lessons Learned Whitepaper. We kindly request any feedback by May 22.

I don't have any more comments.

Thank you to all who provided additional feedback. We will post the finalized published whitepaper on June 4th.

Thank you to everyone for their continued patience during the creation and growth of the Lessons Learned White Paper. We are grateful for all the CAs that participated and the commenters that assisted us through this process. After the final edits and revisions, SSL.com is excited to share the final product with everyone and hope it becomes a useful document to the community when questions should arise about branded SubCAs.

The published White Paper can be viewed and downloaded here: https://www.ssl.com/article/lessons-learned-security-implications-and-good-practices-for-branded-subcas/

We continue to monitor this bug for any further discussions from the community.

I don't see a need to keep this bug open. Shouldn't we close it? If so, I will close it on or about Friday, June 14.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.