Closed Bug 1719916 Opened 3 years ago Closed 2 years ago

SSL.com: Issuance of an EV TLS certificate with incorrect O Field Value

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: support, Assigned: support)

Details

(Whiteboard: [ca-compliance] [ev-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36

Steps to reproduce:

This is a preliminary incident report. Our investigation into this matter is ongoing.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The issue was reported by a third party to SSL.com Support via email and then to our Security Auditors via an internal ticket, in accordance with our Incident Management Policy.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-11-12T15:29:52-06:00 Two-person validation completed to issue an EV TLS certificate to ansonnet.com
2020-11-16T22:55:45+00:00 Certificate with S/N 10975D2ED70F692430714C4586E8BD78 issued to ansonnet.com
2021-03-11 Removal of an auto-population feature of the organization validation system which increased the probability of a human mistake.
2021-05-24 Validation of the organization information of TLS certificates transferred to our new Validation Control System (this VCP was already in use for Code Signing certificates).
2021-07-06T18:12+00:00 Front desk received an email from a third party reporting possible mis-issuance of the above certificate with S/N 10975D2ED70F692430714C4586E8BD78
2021-07-06T20:03+00:00 Event ticket opened by the Validation Specialists (VS) to report the issue.
2021-07-06T20:04+00:00 VS report the issue to Security Auditors, investigation initiated.
2021-07-06T20:21+00:00 Event ticket reviewed by a Security Auditor who requests more information and possible evidence to support the selected "O” field value. Ongoing investigation by VS to provide more information and update the ticket accordingly.
2021-07-06T21:12+00:00 A second Security Auditor engages to assist and discuss the preliminary findings with the VS. The Security Auditor reports internally a possible match with criterion no.8 of our CP/CPS “4.9.1.1 Reasons for Revoking a Subscriber Certificate” which requires revocation within 5 days (preferably 24h) if “SSL.com determines or is made aware that any of the information appearing in the Certificate is inaccurate;”
2021-07-06T21:41+00:00 An internal meeting takes with the participation of VS and the Security Auditor. The issue is confirmed and a list of immediate actions is decided. These include:
(a) reissuance of the certificate with the correct "O" field value
(b) contact the customer regarding revocation of the affected certificate
(c) revocation of the affected certificate before 2021-07-08T17:00+00:00
(d) update the reporter on the issue
(e) continue collection of information to fully document the issue.
2021-07-06T22:13+00:00 VS continue to update the ticket with additional information (action item e)
2021-07-07T01:20+00:00 Our front desk updates the reporter about the issue (action item d)
2021-07-07T04:36+00:00 Our front desk informs the subscriber about the certificate replacement (action item b).
2021-07-07T14:36+00:00 Security Auditors start drafting a preliminary incident report.
2021-07-07T16:57+00:00 Event ticket escalated to an incident.
2021-07-08Τ12:43+00:00 Security Auditors are notified that the customer has initiated certificate replacement (action item a). Revocation targeted for the same date (action item c).
2021-07-08Τ17:01:42+00:00 Certificate revoked
2021-07-08 to 2021-07-09: Security Auditors update this preliminary report after gathering and reviewing all the necessary information.
2021-07-09: Filed initial Bugzilla report (this document).

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The certificate in question was issued in November 2020. In 2021, as part of our ongoing improvement process, we introduced specific technical measures and applied improved validation control systems which minimize the possibility of human mistake when validating organization information. In particular:

  • On 2021-03-11, we removed a productivity feature which was determined to be error prone when processing organization information for larger volumes of certificates.
  • On 2021-05-24 we also transferred the validation of the organization information of TLS certificates under our new Validation Control Panel (VCP) which uses structured online checklists to guide VS through each step of the process.

Since then, we have witnessed a significant improvement in our capacity to perform human reviews of certification information. These measures would have prevented this issuance, and no similar issuances can be performed currently.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

1 certificate, issued November 6, 2020.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Impacted certificates:
Serial Number: 10975D2ED70F692430714C4586E8BD78
https://crt.sh/?id=3672951917

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This issue involved a productivity feature in our certificate systems which has been updated since 2021-03-11.
Prior to 2021-03-11 this feature allowed registrant information (including in this case information placed in the "O" field) to be pre-populated before processing by Validation Specialists.
Prior to our feature update, a manual check by multiple personnel was required and expected to capture any discrepancies. In this instance, our manual review failed to note the discrepancy in "O" field information before issuance.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Immediate actions are described in steps 2-4 of this report and include the immediate involvement of our internal auditing department, revocation of the affected certificate after contacting both the subscriber and the reporter, replacement of the certificate and investigation of the issue.
Initial review finds that the validation control system which has been introduced in 2021 would have prevented this issuance. However, this had not been applied to our systems at time of issuance of the certificate in question.
Our investigation is of course ongoing and an analysis is also being conducted to reveal any underlying weaknesses and, according to the results, decide any additional measures and improvements in our systems and processes, so that such occurrences are not repeated in the future.
In addition to that, our Security Auditors have requested review of all our non-expired non-revoked EV and OV TLS certificates issued before the removal of the feature in question on 2021-03-11 to confirm that this was an isolated incident.

A full incident report shall be filed here when our investigation is complete. In the meantime, we will post regular updates.

Assignee: bwilson → support
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

This is an update to report current progress on this issue.

We have completed review of 35 percent of the target population without finding further occurrences of this issue.

Our investigation is ongoing. A full description shall be made here when the investigation is complete. Status updates to this bug will be made weekly until the final report is presented.

This is an update to report current progress on this issue.

We have completed review of the entire target population and we discovered one (1) similar case.

Impacted certificates:

Serial Number: 42aea52595aa415921fcc5c2fbbee58c
Pre-certificate: https://crt.sh/?id=1450554019
Certificate: https://crt.sh/?id=1573734152

Our investigation has been expanded into a full review of the target population, looking for any other possible discrepancies. A full description shall be made here when the investigation is complete.

This is an update to report current progress on this issue.

During this week we have continued our full review of the target population, trying to locate any other discrepancies. Our investigation is ongoing.

Our plan is to complete this effort within the next two weeks and accordingly update this ticket at that time.

Type: defect → task

This is an update to report current progress on this issue.

We have completed our review of the target population and have found no other similar cases. We intend to take this opportunity to extend this review to identify and address any other discrepancies.

As mentioned in bug 1724520, we also intend to strengthen our code and compliance review methodology to improve our ability to anticipate and prevent similar unintended operations.

Our review is ongoing and we shall continue to post regular updates here.

Thanks Chris.

Could you share a bit more technical detail about what was removed in

2021-03-11 Removal of an auto-population feature of the organization validation system which increased the probability of a human mistake.

For example, in the incident report, it's actually quite difficult to discern the "what went wrong". It seems, by luck of key reuse, the situation here is

  • Misissued certificate: O=AnsonNet Corporation
  • Correct certificate: O=Anson Network Limited

Naturally, the above question is trying to understand "Where did the value of AnsonNet Corporation come from", which is similarly related to understanding "What analysis did SSL.com perform"

Comment #2 provides an example of a different certificate, for crewsbankcorp.com, which has

Beyond understanding where the "bad values" came from (e.g. auto-propagated from the CSR? From some other data source), it'd be useful to understand how the review was performed. Were you examining 100% of certificates for corroboration with the QGIS? Or was there some short-hand used (e.g. "value came from CSR?") to reduce that search space?

Flags: needinfo?(support)

(In reply to Ryan Sleevi from comment #5)

Naturally, the above question is trying to understand "Where did the value of AnsonNet Corporation come from", which is similarly related to understanding "What analysis did SSL.com perform"

This value was submitted by the customer in their CSR. The (now-retired) feature would apply CSR content to the registrant information for later review, and therefore relied on manual analysis and correction before issuance.

Were you examining 100% of certificates for corroboration with the QGIS? Or was there some short-hand used (e.g. "value came from CSR?") to reduce that search space?

We performed a manual and cross-checked review of 100% of potentially impacted certificates.

Flags: needinfo?(support)

We are monitoring this bug for further questions.

Flags: needinfo?(support)

This is our final report on this issue.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The issue was reported by a third party to SSL.com Support via email and then to our Security Auditors via an internal ticket, in accordance with our Incident Management Policy.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2020-11-12T15:29:52-06:00 Two-person validation completed to issue an EV TLS certificate to ansonnet.com.

2020-11-16T22:55:45+00:00 Certificate with S/N 10975D2ED70F692430714C4586E8BD78 issued to ansonnet.com.

2021-03-11 Removal of an auto-population feature of the organization validation system which increased the probability of a human mistake.

2021-05-24 Validation of the organization information of TLS certificates transferred to our new Validation Control System (this VCP was already in use for Code Signing certificates).

2021-07-06T18:12+00:00 Front desk received an email from a third party reporting possible mis-issuance of the above certificate with S/N 10975D2ED70F692430714C4586E8BD78.

2021-07-06T20:03+00:00 Event ticket opened by the Validation Specialists (VS) to report the issue.

2021-07-06T20:04+00:00 VS report the issue to Security Auditors, investigation initiated.

2021-07-06T20:21+00:00 Event ticket reviewed by a Security Auditor who requests more information and possible evidence to support the selected "O” field value. Ongoing investigation by VS to provide more information and update the ticket accordingly.

2021-07-06T21:12+00:00 A second Security Auditor engages to assist and discuss the preliminary findings with the VS. The Security Auditor reports internally a possible match with criterion no.8 of our CP/CPS “4.9.1.1 Reasons for Revoking a Subscriber Certificate” which requires revocation within 5 days (preferably 24h) if “SSL.com determines or is made aware that any of the information appearing in the Certificate is inaccurate;”

2021-07-06T21:41+00:00 An internal meeting takes with the participation of VS and the Security Auditor. The issue is confirmed and a list of immediate actions is decided. These include:

(a) reissuance of the certificate with the correct "O" field value
(b) contact the customer regarding revocation of the affected certificate
(c) revocation of the affected certificate before 2021-07-08T17:00+00:00
(d) update the reporter on the issue
(e) continue collection of information to fully document the issue.

2021-07-06T22:13+00:00 VS continue to update the ticket with additional information (action item e).

2021-07-07T01:20+00:00 Our front desk updates the reporter about the issue (action item d).

2021-07-07T04:36+00:00 Our front desk informs the subscriber about the certificate replacement (action item b).

2021-07-07T14:36+00:00 Security Auditors start drafting a preliminary incident report.

2021-07-07T16:57+00:00 Event ticket escalated to an incident.

2021-07-08Τ12:43+00:00 Security Auditors are notified that the customer has initiated certificate replacement (action item a). Revocation targeted for the same date (action item c).

2021-07-08Τ17:01:42+00:00 Certificate revoked.

2021-07-08 to 2021-07-09: Security Auditors update this preliminary report after gathering and reviewing all the necessary information.

2021-07-09T19:50:00+00:00: Filed initial Bugzilla report, with full report to follow pending completion of in-depth review.

2021-07-12: Initiated review of all our non-expired non-revoked EV and OV TLS certificates issued before the removal of the feature in question on 2021-03-11 to discover any similar cases.

2021-07-23: Completed review of the target population and found one similar case (see section 5 of this report). A detailed manual review of 100% of potentially impacted certificates was also initiated looking for any other possible discrepancies.

2021-08-13: The manual and cross-checked review of the target population was completed without finding any other similar cases (apart from the abovementioned two cases).

2021-08-16 to 2021-08-27: The entire incident was analyzed by the compliance department in coordination with the validation and engineering departments to identify the underlying weaknesses and determine any mitigation measures.

2021-08-30: Started drafting the final Bugzilla report.

2021-09-03: Filed final Bugzilla report (this document).

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The affected certificates were both issued before 2021. In 2021, as part of our ongoing improvement process, we introduced specific technical measures and applied improved validation control systems which minimize the possibility of human mistake when validating organization information. In particular:

On 2021-03-11, we removed a productivity feature which was determined to be error prone when processing organization information for larger volumes of certificates (see section 6).

On 2021-05-24 we also transferred the validation of the organization information of TLS certificates under our new Validation Control Panel (VCP) which uses structured online checklists to guide VS through each step of the process.

Since then, we have witnessed a significant improvement in our capacity to perform human reviews of certification information. These measures would have prevented this issuance, and no similar issuances can be performed currently.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

2 certificates, issued between 2019-05-09 and 2020-11-06.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Impacted certificates:

Serial Number: 10975D2ED70F692430714C4586E8BD78

Certificate: https://crt.sh/?id=3672951917

Serial Number: 42aea52595aa415921fcc5c2fbbee58c

Certificate: https://crt.sh/?id=1573734152

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This issue involved a productivity feature in our certificate systems which has been updated since 2021-03-11.

Prior to 2021-03-11 this feature allowed registrant information (including in this case information placed in the "O" field) to be pre-populated using CSR content before processing by Validation Specialists.

Prior to our feature update, a manual check by multiple personnel was required and expected to capture any discrepancies. In these instances, our manual review failed to note the discrepancy in "O" field information before issuance.

In addition to analyzing the underlying weaknesses which led to the issuance of the problematic certificates (see section 7), a separate analysis took place on why the issue was not discovered by our standard monitoring processes. We determined that the issue was not caught by our linting tools because the mis-issuance was an inaccuracy in the validated content, not a certificate content error. It was also not discovered during quarterly certificate reviews (QCR) due to the very low number of cases (two).

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Immediate actions are described in steps 2-4 of this report and include the immediate involvement of our internal auditing department, revocation and replacement of the affected certificates, and investigation of the issue.

Our initial review confirmed that the validation control system which has been introduced in 2021 would have prevented these issuances. However, this had not been applied to our systems at time of issuance of the certificates in question.

Based on the full review of the entire population of all our non-expired non-revoked EV and OV TLS certificates issued before the removal of the feature in question on 2021-03-11, only a very limited number of certificates (two) were affected by the issue.

In accordance with our Incident Management Policy, an analysis of the root cause for this issue was conducted to reveal any underlying weaknesses. The event management team determined that the source of the issue was an older consideration that such a feature (pre-populating the contents of the form that is used by the VS to process the identity information) would facilitate and expedite the validation process and any discrepancies would be spotted by the VS by manual analysis and correction. In practice, the correction of pre-populated values allowed a higher probability of human error than starting from a clean sheet and entering validated values one-by-one. This aspect was pointed out by our validation team and an update to deprecate the feature was delivered as part of our continuous improvement plan in March 2021.

Flags: needinfo?(support)

If there are no questions or other items to address, I will close this on Wed. 17-Nov-2021.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ev-misissuance]
You need to log in before you can comment on or make changes to this bug.