1731939 - GoDaddy: Issued EV Wildcard Certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Brittany Randall]

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36

Comment 1

[Brittany Randall]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

Problem Summary:
We issued one Extended Validation (EV) wildcard certificate on 9/16/2021, which was in violation of the Guidelines for the Issuance and Management of Extended Validation Certificates Version 1.7.8: Section 9.2, “Wildcard certificates are not allowed for EV Certificates”

Discovery Details:
At 09/20/2021 11:04 MST, while working the Extended Validation (EV) certificate request processing queue, one of the Registration Authority (RA) specialists discovered a pending EV wildcard certificate request and escalated to his supervisor (Note: this certificate request was denied on 09/20/2021 16:44 MST and did not result in an issued EV Wildcard certificate)

At 09/20/2021 11:39 MST, after some research to try to understand how the request was in the queue, the RA supervisor escalated this discovery to the PKI development team who began investigating the root cause (i.e. what allowed this kind of a request to make its way into the system for processing?) and to assess impact (i.e. have any EV Wildcard certificates ever been issued?).

At 09/20/2021 13:30 MST, a system query identified one active Extended Validation (EV) wildcard certificate that had been issued on 09/16/2021 18:07 MST.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY HH:MM (Times are all MST)

• 02/14/2019 19:00 – Code was deployed which enabled the system to accept EV wildcard requests
• 09/09/2021 00:33 - EV Certificate for *.backup.velia.net was requested
• 09/16/2021 18:07 – Full EV certificate validation, including domain and organization completed and EV Certificate for *.backup.velia.net was issued.
• 09/20/2021 11:04 – RA discovered a pending wildcard EV certificate request.
• 09/20/2021 11:39 – RA notifies the developers who begin investigation.
• 09/20/2021 13:30 – During the investigation, developers identify one active, EV wildcard certificate that had been issued on 09/16/2021 18:07 MST. Additionally, developers further confirm that no other EV wildcards have been issued.
• 09/20/2021 13:37 – A system change was implemented to ensure an EV wildcard can no longer be requested, therefore issued. Additional testing was performed to verify the fix was operating as expected.
• 9/20/2021 15:15 – Developers communicate incident to the Compliance team
• 9/20/2021 15:28 – A stakeholder meeting was held to continue the investigation which included: confirming understanding of the issue, gathering additional information, assessing impact against requirements, determining the revocation timeline, drafting this incident report, and next steps.
• 09/21/2021 10:02 - EV Certificate for *.backup.velia.net was revoked.
• 09/21/2021 11:00 – A stakeholder meeting was held to follow up on previous day action items (including verification of revocation of impacted certificate), and obtain additional information for incident reporting.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We have deployed a systematic fix to prevent EV wildcard certificates from being issued.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

1 EV certificate issued Sep 17 01:07:16 2021 GMT: https://crt.sh/?id=5234772527

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.**

https://crt.sh/?id=5234772527 (Revoked)

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

On 02/14/2019 19:00 MST a whitelisting feature was introduced which was intended to allow limited number of internal customers the ability to request wildcards as UCC SANs. This feature was put into place prior to productizing the feature externally, but the code did not consider that a request cannot be an EV type.

The combination of the specificity of the whitelist feature and the specific EV wildcard certificate type made this a very rare occurrence, which went undetected until an RA noticed the pending EV Wildcard which they recognized as not allowed in accordance with policy.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Mitigation Strategy:

• M1. System Update (Completed): Disable the whitelisting functionality to fully prevent the system from accepting EV Wildcard requests
• M2. People Training (In progress): Train/coach RA function to reinforce that EV wildcards are not allowed, adding a procedural layer of scrutiny.
• M3. System Update (On Deck): Update the linter to include a rule which will act as a further preventive measure to identify and prevent any of these certificates from deploying.
• M4. Process Update (On Deck): Update our certificate audit process to detect any EV wildcard certificates in the event any have been issued.

Implementation and Future Actions:

• (M1) On 9/20/2021, we deployed a fix which prevents the system from accepting EV wildcard requests which included turning down the whitelist feature. Completed.
• (M2) Re-training/coaching to be provided to the RA department to reinforce policy (i.e. EV wildcards are not allowed). Target completion date: 9/24/2021
• (M4) Update our certificate audit procedures to specifically look for certificates that are of the type EV where there is a wildcard character present in a common name or subject alternate name. Target completion date: 10/31/2021
• (M3) Build a rule into our linter which would detect any certificate requests in the EV wildcard category and prevent them from issuing. Target completion date: 11/30/2021.

Comment 2

[Brittany Randall]

Update on Item 7 listed above from Comment 1. Specific updates bolded below.

Mitigation Strategy:

• M1. System Update (Completed): Disable the whitelisting functionality to fully prevent the system from accepting EV Wildcard requests
• M2. People Training (Completed): Train/coach RA function to reinforce that EV wildcards are not allowed, adding a procedural layer of scrutiny.
• M3. System Update (On Deck): Update the linter to include a rule which will act as a further preventive measure to identify and prevent any of these certificates from deploying.
• M4. Process Update (On Deck): Update our certificate audit process to detect any EV wildcard certificates in the event any have been issued.

Implementation and Future Actions:

• (M1) On 9/20/2021, we deployed a fix which prevents the system from accepting EV wildcard requests which included turning down the whitelist feature. Completed.
• (M2) On 9/22/2021, the RA supervisors communicated the issue during the team’s daily huddle meetings which included reviewing the specific scenario of EV Wildcard issuance. On 9/23/2021, the RA Director confirmed with his supervisors that all members of the team had been briefed and policy reinforced. Completed.
• (M4) Update our certificate audit procedures to specifically look for certificates that are of the type EV where there is a wildcard character present in a common name or subject alternate name. Target completion date: 10/31/2021
• (M3) Build a rule into our linter which would detect any certificate requests in the EV wildcard category and prevent them from issuing. Target completion date: 11/30/2021.

Comment 3

[Ben Wilson]

It seems that the proposed addition of an audit procedure, which looks for a wildcard character in an EV certificate (M4) doesn't prevent misissuance. Could you please explain why you believe that will be a benefit? Is it an automated process that runs after-the-fact?

Wouldn't it be better to commit to better code review, development lifecycle controls, and/or QA processes that check for potential problems? In other words, we want to encourage CAs to take steps that will prevent recurrences of similar situations in the future (rather than just patch bugs as they are discovered).

Aside from that, I'm going to set the next update to 1-Nov-2021, before which we'll expect a progress update on the linter improvement (and the referenced audit procedures).

Comment 4

[Brittany Randall]

Thanks for the feedback and apologies for any confusion!

M1 and M3 are systematic updates that are meant to serve as preventive measures. These are changes to the code as well as the linter which should prevent an EV Wildcard certificate from being issued (i.e. prevent mis-issuance)

M4 is related to our post issuance certificate audit process. We figured since we do this anyway, we could add a testing attribute that would also serve as a detection mechanism for this particular use case. Hopefully we wouldn’t have to ever rely on it, but we figured it would be a good detective check.

Hope this makes sense! Please let me know if you have any additional questions.

Comment 5

[Brittany Randall]

Update on Item 7 listed above from Comment 2. Specific updates bolded below.

Mitigation Strategy:

• M1. System Update (Completed): Disable the whitelisting functionality to fully prevent the system from accepting EV Wildcard requests
• M2. People Training (Completed): Train/coach RA function to reinforce that EV wildcards are not allowed, adding a procedural layer of scrutiny.
• M3. System Update (Completed): Update the linter to include a rule which will act as a further preventive measure to identify and prevent any of these certificates from deploying.
• M4. Process Update (In progress): Update our certificate audit process to detect any EV wildcard certificates in the event any have been issued.

Implementation and Future Actions:

• (M1) On 9/20/2021, we deployed a fix which prevents the system from accepting EV wildcard requests which included turning down the whitelist feature. Completed.
• (M2) On 9/22/2021, the RA supervisors communicated the issue during the team’s daily huddle meetings which included reviewing the specific scenario of EV Wildcard issuance. On 9/23/2021, the RA Director confirmed with his supervisors that all members of the team had been briefed and policy reinforced. Completed.
• (M3) As of 10/7/2021, a rule has been built into our linter which will detect any certificate requests in the EV wildcard category and prevent them from issuing. Completed.
• (M4) Update our certificate audit procedures to specifically look for certificates that are of the type EV where there is a wildcard character present in a common name or subject alternate name. Target completion date: 10/31/2021. Progress update: Meeting scheduled with internal stakeholders for Monday, 10/11/2021, to check in on progress and determine any next steps for adding to the certificate audit process.

Comment 6

[Brittany Randall]

Related to M4, meeting was held internally to review updates to the certificate audit procedures. Still targeting 10/31 for completion of work.

Comment 7

[Brittany Randall]

Update on Item 7 listed above. Specific updates from Comment 5 to present updated below

Mitigation Strategy:

• M1. System Update (Completed): Disable the whitelisting functionality to fully prevent the system from accepting EV Wildcard requests
• M2. People Training (Completed): Train/coach RA function to reinforce that EV wildcards are not allowed, adding a procedural layer of scrutiny.
• M3. System Update (Completed): Update the linter to include a rule which will act as a further preventive measure to identify and prevent any of these certificates from deploying.
• M4. Process Update (Completed): Update our certificate audit process to detect any EV wildcard certificates in the event any have been issued.

Implementation and Future Actions:

• (M1) On 9/20/2021, we deployed a fix which prevents the system from accepting EV wildcard requests which included turning down the whitelist feature. Completed.
• (M2) On 9/22/2021, the RA supervisors communicated the issue during the team’s daily huddle meetings which included reviewing the specific scenario of EV Wildcard issuance. On 9/23/2021, the RA Director confirmed with his supervisors that all members of the team had been briefed and policy reinforced. Completed.
• (M3) As of 10/7/2021, a rule has been built into our linter which will detect any certificate requests in the EV wildcard category and prevent them from issuing. Completed.
• (M4) As of 10/12/2021, an additional testing attribute has been added to the certificate audit process for EV certificate audits which includes verifying that the domain names associated with a certificate are not of a wildcard type. Note for clarity, this is a procedural process update and not a systematic update which involves a manual check that is performed by the SCS Policy Manager who performs these review.

Comment 8

[Brittany Randall]

All mitigation strategy items have been completed as described in comments above. We are continuing to track this bug for any questions/comments from the community.

Comment 9

[Brittany Randall]

If there isn't any objection, we would like to request closing this bug on 11/1/2021. We will continue to monitor for questions/comments from the community.

Comment 10

[Ben Wilson]

I schedule this for closure on or about Friday 30-Oct-2021.

Comment 11

[Dimitris Zacharopoulos]

Note for clarity, this is a procedural process update and not a systematic update which involves a manual check that is performed by the SCS Policy Manager who performs these review.

I am curious why it had to be a "policy" and not a "technical" measure. Adding a filter in the allowed values of the EV TLS Certificate profile for subject:commonName and SAN:dNSName values to only allow a specific set (that does not include the * character), may be a more effective mitigation.

Comment 12

[Brittany Randall]

Apologies for any confusion. I was trying to make it clear that the action taken specifically for mitigation strategy 4 (M4) was limited to procedures.

The most important mitigation step is Mitigation 1 (M1). M1 was a systematic change that prevents the system from accepting a wildcard EV certificate request (I believe this is similar, if not the same as what you were suggesting😊)

In theory, M1 should be enough to prevent the issue from happening again, but we also decided to identify and implement backup control measures (M2, M3, and M4). Essentially, in the event M1 were to fail in the future, M2 could catch it. If M2 failed, M3 would catch that… and so on.

In my attempt to be thorough, I probably made it a little confusing with all the additional information and extra control measures. Hopefully my explanation helped to add clarify that we did, in fact, make a system update that will prevent the system from accepting EV Wildcard Certificate requests in the future.

Comment 13

[Dimitris Zacharopoulos]

Sounds good Brittany, thank you for the clarification :-)