1666872 - SSL.com: Insufficient validation evidence for the localityName attribute of an OV certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Thomas Zermeno]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36

Steps to reproduce:

During our annual audit, it was discovered that the validation evidence for the localityName attribute of a single OV SSL certificate was not satisfactorily retained. This belonged to a batch of certificates issued for the same organization with branches in the same community. Validation of all the other certificates of the batch was conducted according to our CP/CPS and the applicable requirements and expectations.

The affected certificate was revoked within the required time frame and remediation actions were taken after analysis of the issue. Remediation actions included the introduction of a new simplified step-by-step validation panel which prevents finalizing validation until all requirements are specifically verified and mapped to retained evidence. This solution has been reviewed and approved by our internal compliance team and was also successfully demonstrated to our external auditors.

Our investigation into this issue is ongoing and a complete incident report will be filed in the following days following the recommended template.

Comment 1

[Thomas Zermeno]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

The issue was raised by the external auditors during the annual WebTrust audit as part of the certificate testing on OV SSL certificates. It was initially reported on 2020-08-18. During an internal review this was declared as a security event for evaluation under our Incident Management Policy. After additional investigation and several discussions with our auditors (see timeline below), it was declared an incident on 2020-09-23 (and this bug filed). During this period, SSL.com took actions to respond, analyze and remediate the issue.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Timeline:

2020-08-18T10:35-05:00: Locality issue reported by the external auditors.
2020-08-18T10:56-05:00: Compliance personnel informed of the issue. Initial discussions and the early review of the validation evidence indicated that evidence was consulted, but not retained as expected.
2020-08-19 through 2020-08-21: Further research supported finding that evidence supporting issuance existed and was consulted before issuance, but was not properly retained.
2020-08-21T07:58-05:00: Event ticket created under SSL.com Incident Management Policy to document the investigation, analysis and remediation of this issue. The event management team decided to request the revocation of the certificate within the required timeline, engagement with the external auditors, review of the entire OV corpus and further research on the causes and possible solutions with a high priority.
2020-08-21: Discussion initiated with external auditors regarding this issue.
2020-08-21: Review of similar certificates initiated.
2020-08-21T08:06-05:00: The certificate in question was revoked.
2020-08-21 through 2020-09-13: Per initial investigation, remediation/solution proposed, presented and discussed between the subject matter experts, engineers and compliance personnel. Design and development of the proposed solution initiated in accordance with the specifications decided.
2020-08-25: Investigation of the entire OV corpus completed and found no issues with other certificates. In-depth review initiated.
2020-08-27: Full review of validation process for affected certificate completed, again determining that evidence supporting locality was consulted and confirmed before issuance, but full supporting evidence was not retained as required.
2020-09-13: Final QA completed for remediation. Compliance reviewed and gave final approval.
2020-09-14: Remediation solution applied in production.
2020-09-14: Remediation solution demonstrated to external auditors.
2020-09-23: Final assessment with external auditors regarding classification of this finding. Issue declared an incident under our Incident Management Policy.
2020-09-23: Filed initial Bugzilla report, with full report to follow pending completion of in-depth review.
2020-09-24 through 2020-10-08: Re-evaluation of the entire history of the incident by compliance personnel. Initial root cause assessment confirmed. Follow-up actions include (a) ongoing monitoring of the effectiveness of this remediation and (b) separate and additional sampling of certificates processed through the new Validation Control Panel in the upcoming quarterly internal audit.
2020-10-08: Completion of in-depth review of complete corpus of potentially impacted certificates: no other similar cases found.
2020-10-09: Full report filed.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

A complete review of all OV certificates containing locality information confirmed that the certificate noted above is the only item impacted by this issue.

Validation specialists participated in the investigation of the issue from the beginning and thus were alerted in order to avoid similar cases until full remediation of the issue. Permanent measures have now been deployed to prevent similar cases from arising in the future (see section 7 below).

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

This issue affects one certificate, issued Sep 10, 2019 08:28 CDT.

The certificate above contains "Port Charlotte" as the localityName, but the evidence to support this validation decision was not satisfactorily retained in the validation dossier. The certificate belongs to a batch of certificates issued for the same organization (based on “Wauchula”) with branches in the same community (including “Port Charlotte”). All other certificates in this order use “Wauchula” value in the localityName attribute.

SSL.com's CP/CPS 4.9.1.1 states:

"SSL.com should revoke a certificate within 24 hours and must revoke a Certificate within 5 days if one or more of the following occurs:
SSL.com determines or is made aware that any of the information appearing in the Certificate is inaccurate"

We do not deem the locality information inaccurate per se and we were able to retrieve the supporting evidence during the investigation of the issue, but this evidence was not properly retained before issuance. The certificate was therefore revoked at 2020-08-21T08:06-05:00.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Impacted certificate:
SHA256 Fingerprint: 7A:6F:63:D1:9D:6E:1E:79:B6:67:A3:B1:B5:3C:DB:5A:9C:CC:5C:46:44:96:42:FB:EB:5A:B4:1F:E4:AD:9E:0F
SHA1 Fingerprint: 58:64:E8:CE:EB:CA:25:AE:7F:5E:00:C6:AC:11:D9:16:EF:3A:CE:83
MD5 Fingerprint: 51:1E:B9:C4:12:7C:4D:FE:D0:5F:6D:77:1C:7F:93:A6

Pre-certificate:
https://crt.sh/?id=1868297204

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In the issue reported, multiple certificates were issued for the same organization as part of the same order. The retained validation evidence proves that for all certificates, with the exception of the problematic certificate, all validation steps took place in accordance with our CP/CPS, applicable requirements and expectations.

The certificate in question was issued to a community branch location of the organization, for which a manual, secondary lookup was required and performed. However, evidence for this lookup was not satisfactorily retained to document this validation step. During the investigation of the issue, we were able to retrieve the required evidence and confirm that it was not present in the validation dossier.

According to our analysis (part of our Incident Management process), the complexity of our validation system when used to manually handle batch certificate requests with one or more exceptions (with, in this case, a different locality for a community branch) made the difference here, and allowed for the issue to occur. Validation in such composite cases imposed a requirement for manual inspection and handling of exceptions and corner cases by validation specialists to manage the entire validation evidence dossier and ensure proper and complete evidence retention.

This issue was not detected in our quarterly certificate reviews, and our subsequent investigation confirms that this was a unique occurrence not captured in sampling.

Based on our analysis we took action and remediated this issue (see next section).

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Remediation for this issue has been put in place and includes a new Validation Control Panel which safeguards, rationalizes and streamlines the validation process. In particular, the new system presents applicable validation requirements as a linear list of separate validation tasks, each of which requires retaining validation evidence before completion. The entire process is allowed to be completed only after all mandatory validation actions are completed, with additional final approval steps (including dual control when applicable) of the entire dossier.

Analysis of the root cause for this issue was conducted by the event management team with a focus on the weaknesses which allowed improper retention of validation evidence. The body of certificates was also analyzed in order to reveal any other possible root cause. The event management team determined that specific improvements in the validation interface would have prevented any such occurrences, and worked with software engineers to design a new Validation Control Panel to both rationalize the validation process and enforce the retention of validation evidence.

The new system was designed according to our change management process. This included the identification of requirements and software specifications with the involvement of our validation specialists, software engineers and compliance personnel. A demo was successfully presented to the external auditors to receive feedback and possible recommendations. The system passed QA, UAT and final review by compliance personnel before being approved for production.

Follow-up actions include (a) ongoing monitoring of the effectiveness of this remediation and (b) separate and additional sampling of certificates processed through the new Validation Control Panel in the upcoming quarterly internal audit.

Comment 2

[Ben Wilson]

I'll close this bug on or about 30-October-2020 unless there are additional or unresolved issues still to discuss.