Closed Bug 1838421 Opened 1 year ago Closed 7 months ago

Buypass: Domain validation method using not allowed domain contact

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mads.henriksveen, Assigned: mads.henriksveen)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

No description provided.

Buypass: Domain validation using not allowed domain contact
This is an incident report for one TLS certificate issued by Buypass based on a domain validation using an illegal domain contact. The validation specialist intended to use Method 13 DNS CAA Email Contact (using the contactemail property),but used the email address in the CAA iodef property.

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    Buypass became aware of the problem on June 13, 2023, immediately after issuance

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    All times are local Norwegian times (CET).
    2023-06-13, 13:37: The validation specialist selected a wrong email address from DNS CAA
    2023-06-13, 13:42: An email with an RV was sent to the wrong domain contact
    2023-06-13, 14:25: A confirming response was received
    2023-06-13, 15:00: The affected certificate was issued.
    2023-06-13, 15:45: The validation specialist consulted a team lead to verify the validation and they discovered that an illegal domain contact was used.
    2023-06-13, 16:00: The affected certificate was revoked.

  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    The problem was that the validation specialist misunderstood the use of Method 13 and picked a wrong email address as domain contact. This method is rarely used by Buypass and this was the first time this happened. We have no reason to believe that this is a common mistake done by validations specialists in Buypass. We consider this to be a one-time incident only.

  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    A single certificate was issued before the validation specialist discovered the problem.

  5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.spreadsheet, with one list per distinct problem.
    The affected certificate is https://crt.sh/?id=9642626212

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This was a human error performed by an experienced validation specialist using a domain validation method rarely used. The validation specialist did not follow our internal procedure and picked the only email address present in DNS CAA from the iodef property tag.
After issuance, the validation specialist consulted the team lead to verify the domain validation since this was a method rarely used. The certificate was then revoked immediately after discovering the mistake.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Buypass does already perform a significant amount of domain validations automatically and are working actively to reduce the amount of manual domain validations. However, we still need to perform some manual domain validations that depends on a proper understanding by the validation specialist.
All validation specialists have had extra training on validation Method 13 after this incident and the procedures have been updated to explicitly specifying that DNS CAA iodef property is NOT an allowed domain contact.
We do also focus on this in our internal audits.

Assignee: nobody → mads.henriksveen
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [dv-misissuance]
  1. How often do you perform manual domain validations?
  2. Do you manually perform any domain validation methods besides method 13?
  3. Please describe the process in place at the time of this incident for manually performing method 13. How did the validation specialist look up the CAA record? Was there any programmatic validation performed on the email address selected by the validation specialist? Was the validation specialist also responsible for generating the Random Value and sending it in an email?
  4. What records have you kept for validations performed with method 13?
Flags: needinfo?(mads.henriksveen)
  1. We do manual domain validations weekly, this is done for less than 0,5% of certificates issued
  2. We do also use methods 2, 4 and 7 for manual domain validations (although these methods are mostly used automatically)
  3. We have an internal procedure (in Norwegian) describing how to perform manual validation for method 13:
    a. The CAA record lookup is done using external DNS tools
    b. The CAA contact email address is copied from the tool and pasted into the email, i.e. no programmatic validation is performed on the email address selected
    c. The random value is generated by an internal tool and included in the email and sent, i.e. the validation specialist is not responsible for generating the random value
    d. When a confirming response is received, the validation specialist verifies that this includes the correct random value
  4. Records kept are screenshots from the CAA record lookup, the actual email sent and received, which validation specialist performed the manual validation and date/time of the validation
Flags: needinfo?(mads.henriksveen)

  1. What do you mean by "external DNS tool"? Is it a website operated by a third party, such as https://toolbox.googleapps.com/apps/dig/ ?

  2. Do you also use an external DNS tool when method 7 is performed manually?

  3. Have you reviewed the screenshots of past validations to confirm that they were performed correctly?

Flags: needinfo?(mads.henriksveen)
  1. Dig is one of the tools we are using
  2. Yes, we use the same tools when performing method 7 manually
  3. We have reviewed the evidence for past validations and they were correct
Flags: needinfo?(mads.henriksveen)

To be clear, you mean the dig command line tool running on your own systems? When you say "external", do you mean that the tools are externally-operated (e.g. a website-based tool), or just externally-developed?

Flags: needinfo?(mads.henriksveen)

Sorry for the confusion, we use externally-operated tools like https://toolbox.googleapps.com/apps/dig/.

Flags: needinfo?(mads.henriksveen)

Thanks for providing the clarifications.

I have two big concerns about this incident.

First, externally-operated DNS tools are Delegated Third Parties, and CAs are not allowed to use them to fulfill the requirements of BR Section 3.2.2.4. Thus, every certificate which was validated using one of these tools is misissued and must be revoked within 24 hours. See previously: Bug 1670337, Bug 1651026

Second, Comment 1 contains textbook examples of unacceptable answers. From https://www.ccadb.org/cas/incident-report#incident-reports:

For example, it’s not sufficient to say that “human error” or “lack of training” was a root cause for the incident, nor that “training has been improved” as a solution. While a lack of training may have contributed to the issue, it’s also possible that error-prone tools or practices were required, and making those tools less reliant on training is the correct solution. When training or a process is improved, the CA Owner is expected to provide specific details about the original and corrected material, and specifically detail the changes that were made, and how they tie to the issue. Training alone should not be seen as a sufficient mitigation, and focus should be made on removing error-prone manual steps from the system entirely.

I think that to properly remediate this incident, Buypass must:

  1. Immediately cease performing manual domain validations.
  2. Revoke all certificates whose validations were performed using external DNS tools.
  3. Commit to developing new procedures which do not require a human operator to perform manual steps.
  4. Post a new incident report which meets the expectations of https://www.ccadb.org/cas/incident-report#incident-reports
Flags: needinfo?(mads.henriksveen)

Thank you for the feedback.

We have stopped using external DNS tools for manual domain validations. We have identified the affected certificates and will revoke them in due time.

We will post a new incident report no later than 20 June 2023.

Flags: needinfo?(mads.henriksveen)

We have registered a new bug with the new incident report - see https://bugzilla.mozilla.org/show_bug.cgi?id=1839305

We don't have any new information for this bug.

Bug https://bugzilla.mozilla.org/show_bug.cgi?id=1839305 has been updated today.

We don't have any new information for this bug.

Bug https://bugzilla.mozilla.org/show_bug.cgi?id=1839305 has been updated today.

Is any more information needed for this bug, or could this be closed?

Is any more information needed for this bug, or could this be closed?

The remediation in Bug 1839305 should also address the root cause of this incident, so I think this bug should depend on Bug 1839305 and have the same next update (2023-10-01).

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [dv-misissuance] → [ca-compliance] [dv-misissuance] Next update 2023-10-01
Whiteboard: [ca-compliance] [dv-misissuance] Next update 2023-10-01 → [ca-compliance] [dv-misissuance] Next update 2023-12-04

We have no new information in this bug.

We have no new information in this bug.

We have no new information in this bug.

We have no new information in this bug.

Whiteboard: [ca-compliance] [dv-misissuance] Next update 2023-12-04 → [ca-compliance] [dv-misissuance]

We have no new information in this bug.

We have no new information in this bug.

We have no new information in this bug.

If there are no further comments on questions on this bug, I suggest we close it.

I'll close this tomorrow, 20-Mar-2024, unless there are any objections.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Summary: Buypass: Domain validation using not allowed domain contact → Buypass: Domain validation method using not allowed domain contact
You need to log in before you can comment on or make changes to this bug.