1790693 - SSL.com: Issuance of 1 EV TLS certificate using a Registration/Incorporation Agency not included in our approved public list.

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Thomas Zermeno]

Steps to reproduce:

This is a preliminary incident report. Our investigation into this matter is ongoing.

1. DESCRIPTION (How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.)

During the Q2 2022 internal Quarterly Certificate Review ("QCR") we discovered that one EV TLS certificate was issued using an Incorporating Agency (“IA”) which was not disclosed in our List of Approved Incorporating and Registration Agencies ("Approved List") at the time of issuance.

2. TIMELINE (A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.)

o 2020-06-24 - Ballot SC30 - Disclosure of Registration/Incorporating Agency passes.
o 2020-08-20 - SC30 - Disclosure of Registration/Incorporating Agency in effect, w/deadline for public agency disclosure set for 2020-10-01.
o 2020-09-30 – Initial solution introduced (flat list, periodically updated: https://www.ssl.com/download/list-of-approved-registration-incorporating-agencies/)
o 2021-09-30 – Current solution adopted (live online list: https://secure.ssl.com/registration-authority/approved-registration-and-incorporating-agencies/1)
o 2022-06-13 13:19 CDT - CSR submitted for the EV TLS certificate in question.
o 2022-06-20 10:20 CDT - Evidence for legal existence entered into our Validation Control Panel (“VCP”) using an IA not present in the Approved List.
o 2022-06-20T15:35+00:00 - Validation documentation reviewed.
o 2022-06-20T22:07+00:00 - Validation documentation approved (2-person approval).
o 2022-06-20T22:10+00:00 - Certificate issued.
o 2022-07-01T15:22+00:00 - Q2 2022 QCR initiated.
o 2022-07-21T04:29+00:00 - Certificate in question included in the sample for review per our QCR Procedure.
o 2022-09-11T14:38+00:00 - Certificate reviewed and flagged for discussion of possibly problematic issuance.
o 2022-09-12T15:07+00:00 - The attention of validation teams is drawn to the importance of double-checking that the only legal existence evidence sources that are eligible for use are those disclosed in the Approved List.
o 2022-09-12T18:20+00:00 - Event ticket created and relevant teams notified to review/confirm facts.
o 2022-09-12T18:35+00:00 - Research of all other EV TLS certificates of the QCR sample confirms that they use approved sources only.
o 2022-09-12T22:31+00:00 - Validation team officially confirms facts in the Event ticket.
o 2022-09-13T07:24+00:00 - Issue reviewed by a separate Compliance Officer and escalated to an Incident per our Incident Management Policy. Customer facing teams are notified to inform the customer and proceed with revocation within the required 5-day time frame.
o 2022-09-13 – Filed initial Bugzilla report (this document).

3. IMMEDIATE FIX (Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.)

Policy controls requiring the exclusive use of IA which are already registered in our Approved List are already in place.
The implementation of technical controls to enforce this is underway (see section 7 of this report).
In the meantime, we have focused the attention of our Validation Specialist teams to enforce the existing policy controls for this issue.

4. IMPACT (In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.)

This issue impacts one TLS server certificate issued on 2022-06-20.
Thus far, no other EV TLS certificates have been found to rely on validation evidence of undisclosed IA. Our investigation is ongoing (see section 7 of this report).

5. AFFECTED CERTIFICATES (In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.)

Affected certificates:
S/N: 296adf6f0f552580040fec793e2e8703
https://crt.sh/?sha256=BCDB9882B676EDA5ADEBFBC7C8280A89D307F53A756DFD7BEE8B4E9946C1DFDB (Precertificate)

6. WHAT (Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.)

Our initial review indicates that the certificate in question was correctly validated with sufficient and appropriate evidentiary support, and organizational information was confirmed using the following IA:

https://www.creditchina.gov.cn/

However, this IA was not added to our disclosed Approved List before issuance.
This constitutes a violation of the following section of our CP/CPS:

3.2.2 Authentication of organization identity

"Requests for Certificates which include an organization identity shall be verified using the criteria described below. Items to be verified include the legal existence, legal name, assumed name, legal form and requested address of the organization, and the authority of the requesting party shall be confirmed. SSL.com shall inspect any document relied upon for these purposes for alteration or falsification.
Verification of organization identity in any request for an Extended Validation Certificate shall follow the EV verification procedures described in the EV Guidelines. In particular, whenever validation steps of this section require the use of documentation obtained by an Incorporating Agency or Registration Agency, SSL.com uses only agencies included in its approved, at time of issuance, List of Approved Incorporating and Registration Agencies, which is publicly available at https://www.ssl.com/repository. See section 2.2.8."

Initial review indicates that while our present extended validation process includes separate staged review and approval steps, these did not detect the issue. No technical preventative mechanism was in place at the time of issuance to check for an approved source and block issuance if an approved source is not confirmed.

7. REMEDIATION (List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.)

Immediate actions are described in sections 2-5 of this report and include (1) revocation of the impacted certificate, (2) prioritizing the implementation of technical controls to enforce the selection of approved IA, and (3) focusing attention of Validation Specialist team members to observe existing policy controls.
We have also initiated a review of all non-expired non-revoked EV TLS certificates to capture any other similar occurrences.
Implementation of technical controls (requiring selection of an approved IA before proceeding to issuance) is already underway and is expected to be in place by the end of next week.
This section shall be updated based on the analysis of the incident, as required by our Incident Management Policy.

Comment 1

[Ben Wilson]

Thanks for this report. Looking forward to your update.

Comment 2

[Thomas Zermeno]

This is an update to note our progress.

a. Revocation of the affected certificate took place 2022-09-18 per our CP/CPS and within the required timeline.
b. An update in our Validation Control Panel to enforce the use of only approved and disclosed IA sources has been prepared and passed through testing and verification per our SDLC. Deployment to production is scheduled for next week.
c. The target population of non-expired non-revoked TLS certificates for review has been confirmed; our plan is to complete the analysis mid October 2022.

A full incident report shall be filed here when our investigation is complete. In the meantime, we will post weekly updates.

Comment 3

[Thomas Zermeno]

This is an update to note our progress.

Our Validation Control Panel update has now been deployed to production. This update inserts a step requiring Validation Specialists to use only approved and disclosed IA sources before issuance of Extended Validation certificates can occur.

A full incident report shall be filed here when our investigation is complete. We will continue to post weekly updates until that time.

Comment 4

[Thomas Zermeno]

This is an update to note our progress.

We have now completed our analysis of the target population of potentially impacted non-expired non-revoked TLS certificates. No other issues were found.

We currently expect to complete our investigation and file our full incident report by the end of next week

Comment 5

[Thomas Zermeno]

Our investigation into this matter has concluded. This is our final incident report.

1. DESCRIPTION (How your CA first became aware of the problem (e.g., via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.)

During our Q2 2022 internal Quarterly Certificate Review ("QCR") we discovered that one EV TLS certificate was issued using an Incorporating Agency (“IA”) which was not disclosed in our List of Approved Incorporating and Registration Agencies ("Approved List") at the time of issuance.

2. TIMELINE (A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.)

• 2020-06-24 - Ballot SC30 - Disclosure of Registration/Incorporating Agency passes.
• 2020-08-20 - SC30 - Disclosure of Registration/Incorporating Agency in effect, w/deadline for public agency disclosure set for 2020-10-01.
• 2020-09-30 - Initial solution introduced (flat list, periodically updated: https://www.ssl.com/download/list-of-approved-registration-incorporating-agencies/)
• 2021-09-30 - Current solution adopted (live online list: https://secure.ssl.com/registration-authority/approved-registration-and-incorporating-agencies/1)
• 2022-06-13T18:19+00:00 - CSR submitted for the EV TLS certificate in question.
• 2022-06-20T15:20+00:00 - Evidence for legal existence entered into our Validation Control Panel (“VCP”) using an IA not present in the Approved List.
• 2022-06-20T15:35+00:00 - Validation documentation reviewed.
• 2022-06-20T22:07+00:00 - Validation documentation approved (2-person approval).
• 2022-06-20T22:10+00:00 - Certificate issued.
• 2022-07-01T15:22+00:00 - Q2 2022 QCR initiated.
• 2022-07-18T12:55CT+00:00 - Creation of update request ticket for enforcing use of approved IA in our Validation Control Panel (“VCP”). (Note: this maps to the eventual remediation of this bug.)
• 2022-07-21T04:29+00:00 - Certificate in question included in the sample for review per our QCR Procedure.
• 2022-09-11T14:38+00:00 - Certificate reviewed and flagged for discussion of possibly problematic issuance.
• 2022-09-12T15:07+00:00 - The attention of validation teams is drawn to the importance of double-checking that the only legal existence evidence sources that are eligible for use are those disclosed in the Approved List.
• 2022-09-12T18:20+00:00 - Event ticket created and relevant teams notified to review/confirm facts.
• 2022-09-12T18:35+00:00 - Research of all other EV TLS certificates of the QCR sample confirms that they use approved sources only.
• 2022-09-12T22:31+00:00 - Validation team officially confirms facts in the Event ticket.
• 2022-09-13T07:24+00:00 - Issue reviewed by a separate Compliance Officer and escalated to an Incident per our Incident Management Policy. Customer facing teams are notified to inform the customer and proceed with revocation within the required 5-day time frame.
• 2022-09-13T19:44+00:00 - Filed initial Bugzilla report.
• 2022-09-18T07:13:50 UTC - Affected certificate revoked (per our CP/CPS and within required timeframe)
• 2022-09-21T11:33+00:00 - Review of full population of potentially impacted EV TLS certificates initiated per our Incident Management Policy (“IMP”).
• 2022-09-23T07:55+00:00 - Software update for this remediation reviewed and approved, in accordance with our SDLC.
• 2022-09-23T18:45+00:00 - Root cause analysis performed with the participation of senior Validation Specialists and Compliance Officers per our IMP.
• 2022-10-04T14:21+00:00 - Update deployed to production.
• 2022-10-06T09:06+00:00 - Certificate review completed, confirming that no other EV TLS certificates impacted by this issue.
• 2022-10-14T17:30+00:00- Delivery of final report (this document).

3. IMMEDIATE FIX (Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.)

Policy controls requiring the exclusive use of IA which are already registered in our Approved List were already in place. As an immediate fix, we focused the attention of our Validation Specialist teams to ensure application of these controls.

In parallel, we prioritized the (existing) request to update our VCP enforcing the use of published IA. This action has been completed (see sections 2 and 7 of this report).

4. IMPACT (In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.)

This issue impacts one TLS server certificate issued on 2022-06-20.

5. AFFECTED CERTIFICATES (In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.)

Affected certificates:
S/N: 296adf6f0f552580040fec793e2e8703
https://crt.sh/?sha256=BCDB9882B676EDA5ADEBFBC7C8280A89D307F53A756DFD7BEE8B4E9946C1DFDB (Precertificate)

6. WHAT (Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.)

Our review confirmed that the certificate in question was correctly validated with sufficient and appropriate evidentiary support, and organizational information was verified using the following IA:

https://www.creditchina.gov.cn/

However, this IA was not added to our disclosed Approved List before issuance.

This constitutes a violation of the following section of our CP/CPS:

3.2.2 Authentication of organization identity
"Requests for Certificates which include an organization identity shall be verified using the criteria described below. Items to be verified include the legal existence, legal name, assumed name, legal form and requested address of the organization, and the authority of the requesting party shall be confirmed. SSL.com shall inspect any document relied upon for these purposes for alteration or falsification.
Verification of organization identity in any request for an Extended Validation Certificate shall follow the EV verification procedures described in the EV Guidelines. In particular, whenever validation steps of this section require the use of documentation obtained by an Incorporating Agency or Registration Agency, SSL.com uses only agencies included in its approved, at time of issuance, List of Approved Incorporating and Registration Agencies, which is publicly available at https://www.ssl.com/repository. See section 2.2.8."

Our review found that while our present extended validation process includes separate staged review and approval steps, these did not detect the issue. No technical preventative mechanism was in place at the time of issuance to check for an approved source and block issuance if an approved source is not confirmed.

7. REMEDIATION (List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.)

Immediate actions are described in sections 2-5 of this report and include (1) revocation of the impacted certificate, (2) prioritizing the implementation of technical controls to enforce the selection of approved IA, and (3) focusing attention of Validation Specialist team members ("VS") to observe existing policy controls.

We have completed a review of all non-expired non-revoked EV TLS certificates and confirmed that no other similar occurrences exist.

Our investigation revealed that the issue was able to occur even though documented validation procedures were in place and the order was reviewed by two separate VS.

In this case policy controls alone proved to be insufficient to prevent this misissuance.

Technical controls enforcing selection of an approved IA before proceeding to issuance have now been deployed.

These technical controls remove the possibility for similar occurrences, and they are aligned with our strategy to replace policy controls with technical ones wherever possible.

All remediation actions have now been completed.

Comment 6

[Thomas Zermeno]

As there has been no further comment regarding this matter, we respectfully request closure of this bug.

Comment 7

[Ben Wilson]

I'll close this on or about Friday, 2023-03-24, unless there are any additional questions or issues to discuss.