Open Bug 1961406 Opened 1 month ago Updated 1 day ago

SSL.com: DCV bypass and issue fake certificates for any MX hostname

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: ragtime_knoll5n, Assigned: rebeccak)

Details

(Whiteboard: [ca-compliance] [dv-misissuance] [external])

Attachments

(1 file)

List of affected certificates
4.33 KB, application/vnd.ms-excel
Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36 Edg/135.0.0.0

Steps to reproduce:

SSL.com failed to conduct accurate domain validation control when utilizing the BR 3.2.2.4.14 DCV method (Email to DNS TXT Contact). It incorrectly marks the hostname of the approver's email address as a verified domain, which is completely erroneous.

Steps to reproduce:

  • Navigate to https://dcv-inspector.com and click "Start Test". You will be redirected to a URL such as https://dcv-inspector.com/test/d2b4eee07de5efcb8598f0586cbf2690.
  • Create a TXT record for the domain _validation-contactemail.d2b4eee07de5efcb8598f0586cbf2690.test.dcv-inspector.com with the value myusername@aliyun.com. Here, aliyun.com is both a cloud provider and an email provider, similar to @Yahoo.com, @Gmail.com, or @iCloud.com.
  • Visit SSL.com and request a certificate for the domain d2b4eee07de5efcb8598f0586cbf2690.test.dcv-inspector.com. Then, select myusername@aliyun.com from the email approvers list.
  • Log in to myusername@aliyun.com, retrieve the email that contains the DCV random value, and finalize the DCV validation process.
  • SSL.com will add the domain name of the email address (the part after the @. in this case, aliyun.com) to your list of verified domains.
  • To obtain certificates for aliyun.com and www.aliyun.com, initiate the certificate request. SSL.com will then issue certificates for both aliyun.com and www.aliyun.com.

Affected Certificates

Actual results:

SSL.com verified and issued aliyun.com.
I'm not administrator、admin、hostmaster、postmaster、or webmaster of aliyun.com. and also, _validation-contactemail with the value of my email is never configured for aliyun.com.
So, this is wrong.

Expected results:

Don't list the email domain into verified domains.

Summary: SSL.com: DCV bypass and issue certificates for any MX hostname → SSL.com: DCV bypass and issue fake certificates for any MX hostname

SSL.com acknowledges this bug report and we are investigating further.

Out of an abundance of caution, we have disabled domain validation method 3.2.2.4.14 that was used in the bug report for all SSL/TLS certificates while we investigate. We will provide a preliminary report on or before 2025-04-21.

Preliminary Incident Report

Summary

  • Incident description: An incorrect implementation of the DCV method specified in the SSL.com CP/CPS, section 3.2.2.4.14 (Email to DNS TXT Contact), resulted in mis-issuing a certificate to the hostname of the approver's email address. The certificate has already been revoked, the relevant DCV record has been invalidated and the DCV method has been disabled until remediation of the issue. After scanning the entire corpus of certificates issued with the above method, we identified ten (10) additional affected certificates that were mis-issued and have now been revoked.

  • Relevant policies: This is a violation of the following CP/CPS clauses:

    • 3.2.2.4 Validation of Domain Authorization or Control: SSL.com shall confirm that, prior to the date of Certificate issuance, SSL.com has validated each Fully-Qualified Domain Name (FQDN) listed in the Certificate using at least one of the methods listed below.

    • 3.2.2.4.14 Email to DNS TXT Contact: SSL.com SHALL confirm the Applicant's control over the FQDN by sending a Random Value via email and then receiving a confirming response utilizing the Random Value. The Random Value MUST be sent to a DNS TXT Record Email Contact for the Authorization Domain Name selected to validate the FQDN. Validation not being done properly as stated in BR 3.2.2.4.14 DCV method (Email to DNS TXT Contact).

  • Source of incident disclosure: Third Party Reported. On 2025-04-18 18:42 UTC, the SSL.com Security Auditor account received an automated email notification from Bugzilla about the registration of this current bug by a security researcher.

We would like to thank the security researcher for raising this issue and providing sufficient and well-structured information. SSL.com is processing this incident with the utmost priority.

SSL.com will post our Full Incident Report on or before 2025-05-02.

Assignee: nobody → rebecca_kelley
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [dv-misissuance] [external]

Hi Rebecca - would you mind already sharing the 10 other affected certificates? Thanks

Assignee: rebecca_kelley → rebeccak

Dear Rebecca,

I concur with Arvid’s comment. It is our hope that SSL.com will disclose other 10 certificates, as the legitimate website owners and the broader community have a right to know if potentially fraudulent or mis-validated certificates were issued for their domain names.

Additionally, I understand that the RA (Registration Authority) designated by SSL.com for Entrust employs the same DCV (Domain Control Validation) method (referenced here: https://api.staging.ssl.com/#/Validation%20%2F%20Domains/fetch_domain_by_id, or search for 3.2.2.4.14 via the API definition JSON source at https://api.staging.ssl.com/docs/openapi).

Could you clarify whether the Entrust RA was included in the scope of the preliminary investigation outlined in the report?

Thank you.

(In reply to Arvid Vermote from comment #4)

Hi Rebecca - would you mind already sharing the 10 other affected certificates? Thanks

We understand the concern of the community, below is a list of all affected certificates (the original reported certificate and the 10 additional identified certificates)

https://crt.sh/?id=17926238129
https://crt.sh/?id=13285324095
https://crt.sh/?id=16304469933
https://crt.sh/?id=16304470394
https://crt.sh/?id=16452546552
https://crt.sh/?id=17063971585
https://crt.sh/?id=17489716504
https://crt.sh/?id=17490142956
https://crt.sh/?id=17490213260
https://crt.sh/?id=17490443094
https://crt.sh/?id=17508813791

Full certificate details will be provided per CCADB guidelines in our Full Incident Report

(In reply to Sec Reporter from comment #5)

Dear Rebecca,

I concur with Arvid’s comment. It is our hope that SSL.com will disclose other 10 certificates, as the legitimate website owners and the broader community have a right to know if potentially fraudulent or mis-validated certificates were issued for their domain names.

SSL.com understands the community's concerns and has disclosed the total population of affected/revoked certificates in comment 6. Historical evidence shows that, with the exception of one certificate (https://crt.sh/?id=16452546552), SSL.com did issue previous certificates using compliant DCV evidence during the initial issuance of the certificates which point to non-fraudulent mis-issuances. Unfortunately, upon renewal/reissuance of said certificates, it appears the affected certificates were issued based on invalid DCV evidence and were revoked immediately within 24 hours of being identified.

Additionally, I understand that the RA (Registration Authority) designated by SSL.com for Entrust employs the same DCV (Domain Control Validation) method (referenced here: https://api.staging.ssl.com/#/Validation%20%2F%20Domains/fetch_domain_by_id, or search for 3.2.2.4.14 via the API definition JSON source at https://api.staging.ssl.com/docs/openapi).

Could you clarify whether the Entrust RA was included in the scope of the preliminary investigation outlined in the report?

Thank you for raising this concern. During our investigation we determined that this did not affect the systems and APIs used by Entrust. SSL.com will maintain transparency with the community as we continue our investigation and will provide more information as we complete our Root Cause Analysis.

Thank you.

while not directly included to this bug I think 3.2.2.4.13(email to CAA contact) could have same class of bug. Could you check about those too?

Seo Suchan,

SSL.com does NOT implement 3.2.2.4.13(email to CAA contact) from my evaluation.

Regards.

David Zhao / Senior Researcher
CitadelCore Cyber Security Team

(In reply to CitadelCore Cyber Security Team from comment #9)

Seo Suchan,

SSL.com does NOT implement 3.2.2.4.13(email to CAA contact) from my evaluation.

Regards.

David Zhao / Senior Researcher
CitadelCore Cyber Security Team

that's weird. https://www.ssl.com/faqs/ssl-dv-validation-requirements/#email have mention about using CAA/TXT to set allowed email address.
Is their FAQ out of date?

There are also options to validate any of the above acceptable email addresses through DNS CAA and DNS TXT. For these two options, the DNS records must be properly configured as follows:

DNS CAA Record Example (TTL may vary):
exampledomain.com. 3600 CAA 0 contactemail “hostmaster@exampledomain.com”
DNS TXT Record Example (TTL may vary):
_validation-contactemail.exampledomain.com 3600 TXT “hostmaster@exampledomain.com”

Seo Suchan,

You can register an account from here https://secure.SSL.com/account, and initialize a new evaluation session, they have free ssl to test.

Maybe it's my testing domain configuration does not meet requirements or somehow.

Regards.

David Zhao / Senior Researcher
CitadelCore Cyber Security Team

The CPS says SSL.com supports email to the CCA contact:
https://legal.ssl.com/documents/SSLcom-CP-CPS.pdf

(In reply to Seo Suchan from comment #8)

while not directly included to this bug I think 3.2.2.4.13(email to CAA contact) could have same class of bug. Could you check about those too?

Thank you for the question.

Although we do support 3.2.2.4.13 (Email to CAA contact) through manual validation, we have not automated this process yet. The only automated email processes we actively support are 3.2.2.4.4 (Constructed Email to Domain Contact) and 3.2.2.4.14 (Email to DNS TXT Contact).

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A002038

  • Incident description: An incorrect implementation of the DCV method specified in the SSL.com CP/CPS, section 3.2.2.4.14 (Email to DNS TXT Contact) resulted in the mis-issuance of a DV TLS certificate. Further investigation revealed that the bug also affected the DCV method of section 3.2.2.4.2 (Email, Fax, SMS, or Postal Mail to Domain Contact), and that there were another 10 DV TLS non-fraudulent mis-issuances.

  • Timeline summary:

    • Non-compliance start date: 2024-02-12

    • Non-compliance identified date: 2025-04-18

    • Non-compliance end date: 2025-04-18

  • Relevant policies: This is a violation of the following CP/CPS clauses:

    • 3.2.2.4 Validation of Domain Authorization or Control: SSL.com shall confirm that, prior to the date of Certificate issuance, SSL.com has validated each Fully Qualified Domain Name (FQDN) listed in the Certificate using at least one of the methods listed below.

    • 3.2.2.4.2 Email, Fax, SMS, or Postal Mail to Domain Contact: SSL.com shall confirm the Applicant’s control over the FQDN by sending a Random Value via email, fax, SMS, or postal mail and then receiving a confirming response utilizing the Random Value. The Random Value MUST be sent to an email address, fax/SMS number, or postal mail address identified as a Domain Contact.

    • 3.2.2.4.14 Email to DNS TXT Contact: SSL.com SHALL confirm the Applicant's control over the FQDN by sending a Random Value via email and then receiving a confirming response utilizing the Random Value. The Random Value MUST be sent to a DNS TXT Record Email Contact for the Authorization Domain Name selected to validate the FQDN. Validation not being done properly as stated in BR 3.2.2.4.14 DCV method (Email to DNS TXT Contact).

  • Source of incident disclosure: Third Party Reported. On 2025-04-18 18:42 UTC, the SSL.com Security Auditor account received an automated email notification from Bugzilla about the registration of this current bug by a security researcher.

Impact

  • Total number of certificates: 11

  • Total number of "remaining valid" certificates: 0

  • Affected certificate types: DV TLS

  • Incident heuristic: The full corpus of affected certificates is disclosed in the Appendix.

  • Was issuance stopped in response to this incident, and why or why not?: Issuance using validation 3.2.2.4.14 (Email to DNS TXT Contact) was disabled within 2 hours of the incident being reported.

  • Analysis: N/A - No revocation delay.

  • Additional considerations: Although this bug also affected our implementation of 3.2.2.4.2 (Email, Fax, SMS, or Postal Mail to Domain Contact), SSL.com deprecated this method back on 2024-12-02.

Timeline

All times are in UTC.

2021-10-27:

  • As part of validation method 3.2.2.4.4 (Constructed Email to Domain Contact), A method was introduced that extracted the domain from the approver’s email address.

2024-02-12:

  • A callback was added that invoked a method which introduced a bug to allow potential incorrect domain control validation to be recorded.

2024-12-02:

  • Validation method 3.2.2.4.2 (Email, Fax, SMS, or Postal Mail to Domain Contact) was disabled and validation method 3.2.2.4.14 (Email to DNS TXT Contact) was enabled.

2025-04-18:

  • 18:42: Initial bug report was filed by 3rd party.

  • 20:33: All previous domain validation reuse was nullified for the reported domain.

  • 20:35: The initial code that caused the issue was identified.

  • 20:38: Validation method 3.2.2.4.14 was disabled (comment 2) and it was determined that this did not affect the systems and APIs used by Entrust.

  • 21:16: Reported certificate was revoked.

  • 21:57: Subscriber of revoked certificate was notified via email.

  • 22:03: SSL.com places a comment on bug 1961406 to acknowledge the report.

  • 23:50: Internal ticket registered by the Compliance Business Unit (CBU) in accordance with our Incident Management Policy, confirms this as a violation of section 3.2.2.4 of our CPS and declares an incident.

2025-04-21:

  • 06:39: Reported domain was added to high-risk list.

  • 14:30: All reusable domain validations tied to validation method 3.2.2.4.2 and 3.2.2.4.14 that were deemed improper were nullified.

  • 17:07: Investigation completed in determining the full population of certificates that relied on validation method 3.2.2.4.2 and 3.2.2.4.14.

  • 18:29: An additional ten (10) mis-issued certificates were identified, revoked, and subscribers were notified.

  • 18:40 Preliminary Incident Report posted to Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=1961406#c3)

2025-04-22 to 2025-05-01:

  • A detailed investigation/collection of evidence surrounding the events that led to this bug was conducted along with a Root Cause Analysis and discussion of possible remediation actions.

2025-05-02:

  • Submission of the Final Incident Report (this report)

Related Incidents

No prior related incidents with same root cause.

Root Cause Analysis

Contributing Factor #1: Email domain treated as a verified domain when processing Domain Control Validation (DCV)

  • Description:

    An architectural oversight in SSL.com’s domain validation system led to misinterpretation of domain ownership when processing Domain Control Validation (DCV) requests using certain email-based methods, including BR 3.2.2.4.14 (Email to DNS TXT Contact) and BR 3.2.2.4.2 (Email to Domain Contact). The scope of the issue was limited to a small subset of SSL.com’s validation APIs and systems. It did not impact our enterprise platforms, certificate lifecycle management integrations, or partner-reserved systems. This is further supported by the small population of affected customers and certificates.

    On 2021-10-27, a method was introduced that extracted the domain portion of the approver's email address and used it to generate domain control validation records. This logic was originally designed and used for BR 3.2.2.4.4 (Constructed Email to Domain Contact), where the domain in the email address (e.g., admin@example.com) matches the domain being validated. In that limited context, the behavior was valid and functioned as intended.

    On 2024-02-12, a process was added that linked domain validation records to internal system objects. As a result, the previously introduced logic caused the system to propagate the email domain (e.g., @aliyun.com) instead of the subscriber-requested domain (e.g., example.com). In the case of BR 3.2.2.4.2 (Email to Domain Contact), this led to mis-issuance in situations where WHOIS records contained email addresses with domains that did not match the domain being validated. However, the scope of impact was limited due to the relatively low adoption of this method and the narrow set of conditions required for the flaw to occur, specifically instances where the WHOIS contact email domain differed from the domain under validation.

    On 2024-12-02, validation method BR 3.2.2.4.14 was enabled. Because this method requires domain control to be demonstrated via DNS TXT records under the subscriber’s requested domain, not the approver’s email domain, the legacy behavior caused the system to treat the wrong domain as validated. This further contributed to erroneous certificate issuance for domains not under the subscriber's control, such as aliyun.com.

    Further investigation confirmed that BR 3.2.2.4.2 (Email to Domain Contact), which was deprecated on 2024-12-02, was also impacted during the 2024-02-12 to 2024-12-02, window. The issue was confined to these two methods, and no other validation methods or certificate types were affected.

    SSL.com has temporarily disabled method 3.2.2.4.14, revoked the affected certificates, invalidated the related DCV records, and is implementing additional safeguards to prevent and mitigate any recurrence (outlined below in our Action Items). We remain committed to transparency and accountability in addressing this oversight.

  • Timeline:

  • 2021-10-27: As part of validation method 3.2.2.4.4 (Constructed Email to Domain Contact), A method was introduced that extracted the domain from the approver’s email address.

  • 2024-02-12: A callback was added that invoked a method which introduced a bug to allow potential incorrect domain control validation to be recorded.

  • 2024-12-02: Validation method 3.2.2.4.2 was disabled and validation method 3.2.2.4.14 was enabled.

  • 2025-04-18 18:42: Initial bug report was filed by 3rd party to notify us of mis-issuance due to invalid domain control validation

  • 2025-04-18 20:04: Validation method 3.2.2.4.14 was disabled.

  • 2025-04-21 14:30: All reusable domain validations tied to validation method 3.2.2.4.2 and 3.2.2.4.14 that were deemed improper were nullified.

  • Detection: This bug went undetected in production until it was externally reported, because the flaw was introduced outside of the DCV implementation itself. It was not included in our case testing, or method-specific validation checks. The populate references callback, while seemingly unrelated to DCV, mutated critical validation fields in ways that directly impacted domain control enforcement.

  • Interaction with other factors: N/A

  • Root Cause Analysis methodology used: 5-Whys

Lessons Learned

  • What went well:

  • The issue was promptly escalated within minutes of the initial third-party bug report, allowing the engineering teams to quickly assess the potential impact and disable the affected validation method.

  • What didn’t go well:

  • An incorrect implementation in one of our systems affected DCV 3.2.2.4.2 and 3.2.2.4.14 methods, resulting in the issuance of certificates to non-validated domains.

  • Existing test suites lacked edge case scenarios covering mixed domain contexts (email domain ≠ validation domain), allowing the regression to persist undetected.

  • Where we got lucky:

  • The total number of affection certificates was limited to eleven (11).

  • The other affected validation method 3.2.2.4.2 had been disabled since 2024-12-02.

  • Based on our historical records, with the exception of the certificate issued by the security researcher, none of the mis-issuances were fraudulent.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Disable validation method 3.2.2.4.14 Prevent Contributing Factor #1 Testing behavior in staging and production environment. Periodic checks in the DCV DB to ensure no re-occurrence. - Completed
Deploy a patch to correct the 3.2.2.4.14 domain validation method. Prevent Contributing Factor #1 Testing behavior in staging and production environment. Quarterly certificate reviews. External audits 2025-05-09 In progress
Expand test coverage to include edge case scenarios where the approver’s email domain differs from the domain under validation. This includes implementing new unit and integration tests specifically for email-based validation methods (e.g., 3.2.2.4.14) Mitigate, Detect Contributing Factor #1 Code Test Coverage analysis 2025-05-23 In progress

Appendix

affected-certificates.csv

Rebecca,

Based on SSL.com’s full incident report (Comment #15), prior to enabled of the "Email to DNS TXT contact" method on SSL.com's PKI, the WHOIS-based email validation was also impacted.

My question is: why didn't SSL.com detect the DCV issues when using WHOIS method? This situation is rather prevalent, as it is common for the hostname of a domain's WHOIS contact to differ from the domain itself(before a domain name’s creation there’s no email hosted on it). Your testing scenarios should have the coverage to that.

Thank you.

David Z @CCSTeam

This report appears to continue a pattern of incident reports filed by SSL.com over the past year related to the issuance of publicly-trusted certificates that violated essential security controls defined by the TLS BRs and its own policy documents.

Those other related incidents include:

  • 1938236: SSL.com: Failure to process CAA records from one SubCA
  • 1932973: SSL.com: CAA Empty set handling results in Wildcard issuance
  • 1931615: SSL.com: Entrust API and CAA checking (related to the above)
  • 1927532: SSL.com: Issuance of certificates using keys previously reported as compromised

The frequency and nature of these failures raise concerns regarding SSL.com's operational stability, adherence to policy (i.e., the TLS BRs and its own policies), and the overall effectiveness of its internal controls.

These issues also raise questions related to:

  • gaps in testing and validation processes (both of its own software and that provided by vendors ultimately implemented within SSL.com’s systems/processes),
  • weaknesses in configuration and change management,
  • challenges in standards interpretation and corresponding technical implementation, and
  • dependencies on external software vendors.

Questions:

1) Did SSL.com consider permanently sunsetting use of email-based DCV, rather than patching Method “3.2.2.4.14” as described in the Action Items table? If so, what was the outcome of that consideration? If not, how come?

2) 1927532 included an Action Item to update the CP/CPS Change Management System to require a “thorough analysis of possible implementation-level implications by all key roles.” In practice, what does that look like? If we were to take the issue highlighted in this report as an example, can you walk us through the process of how the Action Item, once implemented, would have prevented this issue?

3) Beyond the action described in (2), what other fundamental, systemic changes has SSL.com implemented or planned across its organizational structure, software development lifecycle, operational procedures, internal control frameworks, and compliance oversight functions in attempt to prevent entire classes of errors like those observed over the past year (e.g., testing escapes, configuration errors, misinterpretations) from recurring?

4) How has SSL.com demonstrably strengthened its internal audit, monitoring, and assurance functions to enable the proactive detection of compliance deviations and operational issues related to DCV and CAA checking before they escalate into incidents? What evidence demonstrates this enhanced capability?

5) How is SSL.com investing in developing and retaining the deep internal technical and policy expertise required to accurately interpret complex standards, and implement robust, compliant processes independently, thereby reducing reliance on reactive corrections following vendor failures or community feedback?

6) What internal and external accountability mechanisms are in place to ensure that the commitments made in incident reports and responses to Root Program inquiries are fulfilled effectively and translate into measurable, lasting improvements in operational performance and compliance posture?

7) What specific enhancements, if any, will SSL.com request of its auditor during future compliance audits regarding scope, methodology, sampling strategy, and technical depth to explicitly target the types of failures observed in this bug (e.g., complex logic errors, configuration validation, edge-case handling, vendor software interactions) - and those others cited as concern at the beginning of this response?

8) Are there broader changes SSL.com would suggest to the TLS BRs or audit frameworks to promote more reliable detection of these issues?

9) It appears that these issues were detected by third-parties. The fact that SSL.com offers free certificates and an accessible platform seems to have played a role in uncovering some of these issues. One possible interpretation of this is that CAs with more open architectures are subjected to a higher degree of continuous, informal (and formal) scrutiny than those that do not. Conversely, CAs operating more opaquely – might harbor similar or potentially worse vulnerabilities that remain undetected for longer periods simply due to reduced external visibility and testing.

How might the community encourage broader availability of CA services to security researchers and other interested parties dedicated to improving web security such that these evaluations can take place more consistently across the ecosystem?

Flags: needinfo?(rebeccak)

(In reply to CitadelCore Cyber Security Team from comment #16)

Rebecca,

Based on SSL.com’s full incident report (Comment #15), prior to enabled of the "Email to DNS TXT contact" method on SSL.com's PKI, the WHOIS-based email validation was also impacted.

My question is: why didn't SSL.com detect the DCV issues when using WHOIS method? This situation is rather prevalent, as it is common for the hostname of a domain's WHOIS contact to differ from the domain itself(before a domain name’s creation there’s no email hosted on it). Your testing scenarios should have the coverage to that.

Hi David -

Thank you for your question and for pointing out an important aspect of the incident.

We agree that robust testing should cover such scenarios however, the core issue here was not in the DCV logic itself, but rather in an external architectural change introduced in 2024-02-12: a callback function that unintentionally linked certificate requests to domain control validations via the email domain of the contact, rather than the requested domain.

This callback was originally introduced for use in a different validation method and unwittingly allowed to influence the behavior of 3.2.2.4.2. Prior to this change, our system correctly enforced domain control, and our audit trail confirms that no mis-issuances occurred before the callback went live.

To your point, this specific flaw escaped detection because the callback operated outside the traditional DCV pipeline and mutated validation fields indirectly. The process did not come from our DCV method logic, and as such, it was not exercised by our existing unit or integration tests that focus on DCV methods.

SSL.com recognizes this gap and has implemented remediation steps, including expanding test coverage specifically for scenarios where the approver’s email domain and the validated domain differ.

Flags: needinfo?(rebeccak)

(In reply to chrome-root-program from comment #17)

We at SSL.com have already begun exploring long term solutions that will help address these concerns and we are currently working on a response to your questions.

Rebecca,

How's the progress on the action items due by 2025-05-09?

David Zhao

Flags: needinfo?(rebeccak)

(In reply to Rebecca Kelley from comment #19)

(In reply to chrome-root-program from comment #17)

We at SSL.com have already begun exploring long term solutions that will help address these concerns and we are currently working on a response to your questions.

The requirements are to provide a timeline for unanswered questions, not just to say you're vaguely working on them.

CA Owners should respond promptly to comments and questions, and MUST respond within 7 days, even if only to acknowledge the request and provide a timeline for a full response.

It would be beneficial to at least provide updates on where you are on creating a response to these questions, otherwise it is hard to gauge when the responses will appear.

(To Wayne from comment #21)

Please keep patience with CA participants, as some of their work may take more time to meet the responsable standards.

However, it is indeed recommended that SSL.com enhance transparency by regularly disclosing the latest progress (e.g., every 7 days) during the specific rectification processes of handling critical incidents.

Perhaps the industry has better measures to enhance transparency of communication?

Regards,

David Zhao / Senior Researcher
CitadelCore Cyber Security Team

This is an update to report our progress with remediation actions and as a response to comment 20.

The following action item has been completed:

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Deploy a patch to correct the 3.2.2.4.14 domain validation method. Prevent Contributing Factor #1 Testing behavior in staging and production environment. Quarterly certificate reviews. External audits 2025-05-09 Completed

SSL.com has deployed the patch to our staging and production environments according to the plan and confirmed that the attack scenario of this bug is addressed. The update is available to test in our staging environment but due to the nature of this incident, we have chosen to err on the side of caution and keep the method disabled in the production environment until at least the completion of our action item no.3 (expanded test coverage, unit and integration tests).

Flags: needinfo?(rebeccak)

(In reply to Wayne from comment #21)

(In reply to Rebecca Kelley from comment #19)

(In reply to chrome-root-program from comment #17)

We at SSL.com have already begun exploring long term solutions that will help address these concerns and we are currently working on a response to your questions.

The requirements are to provide a timeline for unanswered questions, not just to say you're vaguely working on them.

CA Owners should respond promptly to comments and questions, and MUST respond within 7 days, even if only to acknowledge the request and provide a timeline for a full response.

It would be beneficial to at least provide updates on where you are on creating a response to these questions, otherwise it is hard to gauge when the responses will appear.

Hi Wayne,

Due to the nature and depth of questions that were asked, we are taking the time to review, verify, and confirm each written answer with not only the departments they are focused on, but also the leadership of SSL.com. At the time of our acknowledgment in comment 19, we did not have a timeline, as these questions require thorough internal research and transparency that is expected by the Root Store and the community. We plan to have our answers for Chrome on or before 2025-05-16.

(In reply to chrome-root-program from comment #17)

Dear Chrome,

We apologize for the extended time it took us to complete your questions, and we appreciate your and the community's patience, as we answered each question with transparency and reflection.

Please review the answers below.

As always, we are monitoring and available to answer the community if more questions arise.

Questions:

1) Did SSL.com consider permanently sunsetting use of email-based DCV, rather than patching Method “3.2.2.4.14” as described in the Action Items table? If so, what was the outcome of that consideration? If not, how come?

SSL.com considered the scope of the affected validation methods, and it was determined that the issue was not with the DCV method themselves, but with the specific process that linked the validated email addresses with requests. As only 3.2.2.4.2 and 3.2.2.4.14 used this specific process, it was determined that they were the only methods that should be restricted. Of course, 3.2.2.4.2 was already deprecated, leaving only 3.2.2.4.14.

Currently, our staging and productions systems have been patched, but DCV method 3.2.2.4.14 remains disabled in production and it will only be considered for re-activation after extensive testing has shown that the method is properly implemented and fully aligned with the BRs (see Comment 23).

SSL.com has also been considering permanently sunsetting use of email-based DCV, especially in light of the shortening of certificate lifespans and the reduced reuse period for DCV evidence. At this time, we determined that removing email-based validation would negatively affect our user base without sufficient planning, lead time, and clear communication, thus we opted for patching the affected email-based methods and keeping them disabled until they are thoroughly tested. In parallel, we continue to promote the use of automated methods [1] and raise their preparedness of our customers for mass revocations [2].

2) 1927532 included an Action Item to update the CP/CPS Change Management System to require a “thorough analysis of possible implementation-level implications by all key roles.” In practice, what does that look like? If we were to take the issue highlighted in this report as an example, can you walk us through the process of how the Action Item, once implemented, would have prevented this issue?

The action item described in bug 1927532 introduced improvements in our CP/CPS Change Management Procedure to ensure proper propagation of policy changes to our systems and processes. The updated CP/CPS Change Management Procedure requires review and sign-off of each normative policy change in our CPS by all key departments (i.e., CA Engineering, Systems Engineering, Networking, Software Engineering, Validation, and Compliance) to ensure that all relevant teams have reviewed the changes in the CP/CPS, identified possible implications and planned ahead for the timely implementation of these policy changes. The process has been operating since January 29, 2025, and has been shown to be effective in identifying implications of CPS/policy updates to all systems (part of “What didn’t go well” in https://bugzilla.mozilla.org/show_bug.cgi?id=1927532#c6).

Based on the above, the updated CP/CPS Change Management Procedure would not be directly applicable to the testing of the code change. However, it does enable early understanding of updates in requirements by all involved teams, and thus proper design, implementation and testing/verification of changes in our systems. The testing processes themselves are covered by the SDLC and SOP describes in our response to question no.3.

3) Beyond the action described in (2), what other fundamental, systemic changes has SSL.com implemented or planned across its organizational structure, software development lifecycle, operational procedures, internal control frameworks, and compliance oversight functions in attempt to prevent entire classes of errors like those observed over the past year (e.g., testing escapes, configuration errors, misinterpretations) from recurring?

SSL.com has implemented a series of organizational and procedural enhancements that go above our existing Baseline Requirements (BR) compliant frameworks to address systemic issues and improve our compliance posture.

At the organizational and governance level, SSL.com has initiated a SOC 2 certification effort that is already underway. This initiative is driving fundamental, systemic changes across the organization—transforming our software development lifecycle, operational procedures, internal control frameworks, and compliance oversight functions. To support and sustain these changes, we are adopting a Governance, Risk, and Compliance (GRC) tool that integrates WebTrust and SOC 2 frameworks, enabling centralized management of controls, evidence collection, risk assessments, and audit readiness. By embedding security and control rigor into our operations, we are actively working to address entire classes of issues such as testing escapes, configuration errors, and procedural misinterpretations. We expect that these initiatives will improve the robustness of our controls and the alignment with industry standards, in a way that is measurable, auditable, and continuously improving.

At the engineering and operational level, we are transitioning to a domain-based operational team model to consolidate subject matter expertise and promote accountability within clearly defined scopes of responsibility. This model strengthens ownership and institutional knowledge, enabling faster and more accurate decision-making.

We are also expanding and refining our Software Development Lifecycle (SDLC) to introduce additional scrutiny at critical control points. While our current practices meet industry standards, we are reinforcing them with stricter internal QA testing prior to code reviews, more robust integration testing in shared environments, and gated promotions that require both technical and compliance validation. These safeguards are complemented by standardized procedures for unit and integration testing, designed to improve coverage and consistency across our codebase.

In parallel, we are formalizing a comprehensive Standard Operating Procedure (SOP) framework that documents cross-functional responsibilities, internal controls, escalation protocols, and quality expectations. This SOP supports our commitment to operational discipline and provides a common foundation for continual improvement across all teams. Collectively, these efforts represent a deeper investment in preventive quality and a strategic move toward a more resilient and audit-ready engineering organization.

It should also be noted that SSL.com is committed to fostering the intellectual growth of all employees. As described in the answer to Question #5, we encourage enrollment in Continuing Education and other industry related training opportunities.

4) How has SSL.com demonstrably strengthened its internal audit, monitoring, and assurance functions to enable the proactive detection of compliance deviations and operational issues related to DCV and CAA checking before they escalate into incidents? What evidence demonstrates this enhanced capability?

To strengthen our ability to proactively detect compliance deviations and operational issues, particularly related to Domain Control Validation (DCV) and CAA checking, SSL.com is continuously enhancing its internal audit, monitoring, and assurance functions.

We maintain extensive unit test coverage for these critical validation mechanisms as part of our SDLC. Our current SDLC processes include two-person code reviews and pre-deployment QA testing to ensure quality and compliance with industry standards.

We plan to expand these test suites to include additional integration test cases that cover complex edge conditions and compliance-focused scenarios drawn from the Baseline Requirements and recent incident reviews. We also plan to schedule these expanded test cases to run automatically on a regular basis against our staging environments, ensuring that DCV logic is continuously exercised and monitored in a controlled setting. To complement this, we are implementing automated alerting for any deviations from expected behavior, enabling rapid detection and triage by engineering and compliance teams. These improvements provide a tangible and auditable way to strengthen assurance and identify issues should they reach production.

Regarding CAA checking, we believe our analysis provided in Bug #1932973 (and the duplicate Bug 1931615) demonstrates that CAA is verified centrally by SSL.com, at issuance time and is now also enhanced with MPIC. The code implementing the CAA checks has been tested for years in production and it is also used by many public CAs. For several PKI core elements like CA management and CAA, we rely on our Enterprise CA software vendor and maintain a close relationship to convey issues, requests for improvements and notices of changes in the BRs and Root Store Program policies. In several cases we have contributed fixes, improvements and suggestions, that were later included in CA software releases for all CAs. Based on the above-mentioned incident with the empty CAA set for wildcards, we have extended the CAA Test Suite with additional tests to cover the case in question. We are happy to consider any other recommendations from the community.

With the addition of Enterprise CAs that SSL.com has partnered with in the past year, we have had an exponential growth of TLS certificates. Prior to this bug, we have not encountered a DCV issue since 2022-01-17. In total we have fourteen (14) impacted TLS certificates during a 40-month time period. Although it is not ideal to have any compliance deviations, this still accounts for evidence to demonstrate our enhanced capabilities.

5) How is SSL.com investing in developing and retaining the deep internal technical and policy expertise required to accurately interpret complex standards, and implement robust, compliant processes independently, thereby reducing reliance on reactive corrections following vendor failures or community feedback?

At SSL.com, we are committed to maintaining and enhancing our technical and policy expertise to ensure robust, compliant, and independently sound practices. Over the past two years, we have made significant investments in both human resources and process improvements to strengthen our internal capabilities. We have expanded our compliance, development, and PKI teams by hiring qualified professionals and industry veterans to bring valuable experience and knowledge to SSL.com.

We continue to provide professional development for all technical and policy-focused employees. This includes training in the latest compliance requirements, PKI practices, and software updates. By fostering a culture of continuous learning, and actively participating in the industry fora, we reduce our reliance on reactive measures and position our team to proactively address industry changes.

Regarding the referenced CAA-related incidents, it is important to note that the CAA module involved is sourced from a vendor commonly utilized by numerous publicly trusted CAs. SSL.com has been actively collaborating with this vendor to address identified bugs. Our teams have played a crucial role in identifying problems, implementing fixes, and contributing patches back to the provider. These collaborative efforts have resolved the issues on our end and contributed to ecosystem-wide improvements, as the patches were later shared with other CAs. This proactive engagement shows our commitment to maintaining compliance and contributing to the collective security and reliability of the public PKI ecosystem.

By strengthening our internal expertise, improving training, automating critical processes, and engaging in joint improvements within the CA ecosystem, SSL.com is dedicated to maintaining robust compliance practices. We recognize the importance of proactive measures over reactive corrections and continue to invest in our teams to uphold this standard.

6) What internal and external accountability mechanisms are in place to ensure that the commitments made in incident reports and responses to Root Program inquiries are fulfilled effectively and translate into measurable, lasting improvements in operational performance and compliance posture?

SSL.com has implemented a structured framework of internal and external accountability mechanisms to ensure that commitments made in incident reports and responses to Root Program inquiries are effectively fulfilled and result in sustained improvements in operational performance and compliance posture. This framework is structured around three core components: Internal Accountability, External Accountability, and Measurable Improvements.

Internal Accountability Mechanisms:

  • Incident Tracking and Remediation Management: We use a centralized tracking system to document and monitor the implementation of corrective actions arising from incident reports and Root Program inquiries. This system assigns specific responsibilities to relevant departments (e.g., Compliance, Development, PKI Operations) and sets defined timelines for completion.

  • Compliance Oversight Committee: A dedicated internal committee, comprising senior management and key stakeholders from Compliance, Development, and PKI teams, conducts periodic reviews to evaluate incidents, corrective actions and proactive measures that are aligned with the commitments outlined in incident reports.

  • Policy Integration and Process Standardization: Identified corrective actions are systematically integrated into SSL.com's standard operating procedures and technical policies to prevent recurrence. These updates are communicated company-wide and reinforced through regular training sessions.

  • Internal Auditing and Self-Assessment: We conduct periodic internal audits to validate the implementation of corrective actions. This includes spot checks, documentation reviews, and process testing to ensure changes have been effectively applied and are functioning as intended.

    External Accountability Mechanisms:

  • Ongoing Communication with Root Programs: SSL.com maintains open and continuous communication with the Root Programs through Bugzilla providing periodic progress reports, documentation of completed actions, and any requested follow-up information to demonstrate adherence to the commitments made in our incident reports.

  • Independent Audits and Assessments: We engage with external auditors to independently verify our compliance with industry standards, including Baseline Requirements and relevant Root Program policies. This provides objective validation of the effectiveness of implemented corrective actions.

  • Vendor and Partner Collaboration: SSL.com collaborates with third-party vendors and CA community partners to share lessons learned and implement mutually beneficial improvements. For example, in the context of the CAA module vendor issues, our team actively participated in testing and deploying patches that addressed our specific incidents and contributed to broader ecosystem-wide fixes.

  • Community Engagement and Reporting: We actively participate in industry working groups and CA/B Forum discussions to share findings, seek feedback, and align our practices with evolving best practices in compliance and security.

    Measurable Improvements and Continuous Monitoring:

  • Post-Incident Analysis and Reporting: Following the implementation of corrective actions, SSL.com conducts post-incident analyses to assess the effectiveness of changes and identify any areas for further improvement. These findings are documented and reviewed by senior management.

  • Sustained Improvement Program: To ensure that implemented changes are not just reactive, we have launched a sustained improvement program including ongoing training, system monitoring, and periodic policy reviews. This program is designed to reinforce compliance posture and prevent recurrence of similar incidents.

SSL.com remains committed to fostering a culture of continuous improvement, accountability, and proactive risk management. By systematically integrating corrective actions into our policies, conducting rigorous internal and external audits, and maintaining open communication with Root Programs and external auditors, we ensure that our commitments translate into lasting improvements in operational performance and compliance.

7) What specific enhancements, if any, will SSL.com request of its auditor during future compliance audits regarding scope, methodology, sampling strategy, and technical depth to explicitly target the types of failures observed in this bug (e.g., complex logic errors, configuration validation, edge-case handling, vendor software interactions) - and those others cited as concern at the beginning of this response?

Given the low volume of affected certificates in our bugs and the complexity of these kinds of interactions, there may not be specific sampling changes (strategy, size) that would materially raise the chances of the external audit spotting such mis-issuances.

However, there are indeed improvements that could be made to our audit requests and might be beneficial for other CAs as well. Per discussion with our WebTrust auditors, we are considering extending the auditor’s testing by elaborating Change Management provisions in our CP/CPS. This would increase the transparency of the public, and would also enable deeper testing of these processes, giving additional assurance as part of the WTCA report.

Other things that could potentially be considered for inclusion are:

  • a database of test validations with known errors

  • special considerations of risk for changes that impact validation

  • regular updates to risks and considerations based on updated bugs and other publicly known issues

We believe this is a valuable discussion to be held with the auditors and the community to identify good uses of auditing options in providing more assurance to certificate consumers.

8) Are there broader changes SSL.com would suggest to the TLS BRs or audit frameworks to promote more reliable detection of these issues?

Although not a TLS BR change suggestion, one consideration we have would be the generation of a DCV test suite like the commonly used CAA test suite that can be used to complement a CA's own testing procedures. We think the open-source Boulder repository could be used as an input to the positive/negative testing suites needed -at least- for the DCV methods to ensure adequate coverage including cases that CAs have seen collectively, or issues identified in public incidents. The same DCV testing suites could be used by auditors to provide independent evaluation and assurance of the effectiveness of the CA’s DCV processes.

We, at SSL.com, feel this is a great question for the broader community and should be an open discussion at some point.

9) It appears that these issues were detected by third-parties. The fact that SSL.com offers free certificates and an accessible platform seems to have played a role in uncovering some of these issues. One possible interpretation of this is that CAs with more open architectures are subjected to a higher degree of continuous, informal (and formal) scrutiny than those that do not. Conversely, CAs operating more opaquely – might harbor similar or potentially worse vulnerabilities that remain undetected for longer periods simply due to reduced external visibility and testing.

How might the community encourage broader availability of CA services to security researchers and other interested parties dedicated to improving web security such that these evaluations can take place more consistently across the ecosystem?

The free certificates service was not initially created by SSL.com as a means for security researcher testing, however in the light of the recent incident, we came to realize the benefit of a higher scrutiny to increasing the confidence to our publicly trusted services. We understand that opening the doors for anyone to inspect the organization will always be a risky maneuver. It is not something that any CA would want to take on without due reason. On the other hand, this allowed the identification of a bug that, if kept unnoticed, could potentially have a significant impact in the future.

For CAs to be encouraged to adopt open architectures that expose them to a higher scrutiny and thus may result in more incident disclosures, we feel it is imperative that bug comments of these incidents promote a positive environment from the community, focusing on helpful and useful feedback.

(to chrome-root-program from comment #17)

  1. Did SSL.com consider permanently sunsetting use of email-based DCV, rather than patching Method “3.2.2.4.14” as described in the Action Items table? If so, what was the outcome of that consideration? If not, how come?

CitadelCore Cyber Security Team endorses with SSL.com's conclusion to this question, which:

SSL.com considered the scope of the affected validation methods, and it was determined that the issue was not with the DCV method themselves, but with the specific process that linked the validated email addresses with requests. As only 3.2.2.4.2 and 3.2.2.4.14 used this specific process, it was determined that they were the only methods that should be restricted. Of course, 3.2.2.4.2 was already deprecated, leaving only 3.2.2.4.14.

Currently, our staging and productions systems have been patched, but DCV method 3.2.2.4.14 remains disabled in production and it will only be considered for re-activation after extensive testing has shown that the method is properly implemented and fully aligned with the BRs (see Comment 23).

SSL.com has also been considering permanently sunsetting use of email-based DCV, especially in light of the shortening of certificate lifespans and the reduced reuse period for DCV evidence. At this time, we determined that removing email-based validation would negatively affect our user base without sufficient planning, lead time, and clear communication, thus we opted for patching the affected email-based methods and keeping them disabled until they are thoroughly tested...

CitadelCore Cyber Security Team think Chrome Policy team has an over concern for this bug.

Currently, CA has no neccessary to unset use of email-based DCV permanently, this bug is solely due to the negligence of SSL.com developers during the implementation and testing processes, rather than any major flaws in BR's any email verify definiation. And also, this is a wildly using DCV method.

If sunseting a DCV method every time when a bug is reported, this would essentially be a form of disincentive for security researchers.
It would create a moral dilemma and burden.
This discourages us from publicly disclosing issues to the community in the future.

(to Rebecca Kelley from Comment #25)
And extended discuss for Mass revocation topic to SSL.com/Rebecca:

... In parallel, we continue to promote the use of automated methods [1] and raise their preparedness of our customers for mass revocations.

Mass revocations's ultimize solution is ACME's ARI, It is recommended that ACME subscribers conduct daily checks to verify whether their currently installed certificates are scheduled for revocation and replace them as early as possible. This proactive approach ensures compliance with revocation requirements while maintaining continuous operation of SSL-enabled businesses, thereby achieving both regulatory adherence and uninterrupted service without mutual disruptionm, and without actions from human.

I don't now how's the percentag of the ACME clients recognized ARI, but I think any ACME CA must provide ARI as soon as possible(currently SSL.com doesn't supprot ARI: https://acme.ssl.com/sslcom-dv-rsa ).

I hope opinion from my team can help others to read this Case and extend the discussion of related issues.

Regards,

David Zhao / Senior Researcher
CitadelCore Cyber Security Team

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: