1546776 - SecureTrust: Unvalidated domain in certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[fcorday]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During an internal self-audit on April 23rd, 2019 at 14:16 CDT, a certificate was discovered to have been mis-issued. In researching the recent usage of the various supported domain validation methods, an engineer noticed that a certificate had been validated using BR method 3.2.2.4.3. As this method is being deprecated, he decided to investigate further. The certificate was intended to be an internal only, test certificate for a proof of concept for a current, managed service customer. However, the certificate was issued as a public, 30-day trial OV certificate.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

• 04.23.2019, 12:26 CDT self-audit conducted
• 04.23.2019, 14:16 CDT self-audit discovered a validation irregularity
• 04.23.2019, 14:26 CDT conference call conducted with the validation team determined an analyst mistakenly issued a public certificate without proper domain validation
• 04.23.2019, 14:32 CDT certificate revoked

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Yes, Trustwave has corrected the problem. We conducted appropriate training and updated internal documentation to ensure proper internal certificate requests follow protocols to ensure proper domain validation.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

For this proof of concept, 1 certificate was issued on 04.08.2019, 13:25 CDT. The certificate was to establish encrypted communication from a Trustwave box to the customer.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Please see: https://crt.sh/?id=1415815247

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The internal requestor of the certificate did not use proper process and instead relied on other communication means to request the certificate. The validation issue arose when the certificate common name was amended from a Trustwave URL to the customer URL for testing purposes without proper validation. Confusion was introduced based on the customer relationship and which entity possessed authority and control of the testing URL. A gap in knowledge and documentation available to the validation team involving short-lived, test certificates was discovered which would have prevented this instance.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Additional training and proper internal step by step documentation for requests such as these have been provided to our validation team. We may also review/introduce additional programmatic controls based on further analysis.

Comment 1

[Ryan Sleevi]

Frank: Thanks for filing this bug.

I'm rather confused about your answer to #6, and having a hard time understanding a bit about what went wrong. Could you provide additional technical details about what occurred and when it occurred?

Comment 2

[fcorday]

Ryan: I will attempt to clarify.

1. April 5 – Certificate Signing Request (CSR) submitted where the validation agent parsed the CSR for an internal Trustwave requestor for a proof of concept of a managed service customer.
2. April 8 - Requestor asked the validation agent to update the certificate from the Trustwave.com domain to a customer domain name for testing.
3. April 8 - Validation agent updated the domain and validation method (phone validation toggled), and issued the certificate.

The validation agent incorrectly thought this proof of concept was operating in a test environment. For this reason, the validation agent neglected to confirm domain control when updating the validation method, as he believed the test environment was not bound by the same requirements for validation. I would be happy to clarify if you have specific questions.

Comment 3

[Ryan Sleevi]

Thanks! These extra details help provide a clearer understanding. To make sure I'm echoing back correctly what went wrong:

1. An employee of Trustwave created a CSR to demonstrate a proof of concept of what a managed service customer would experience, and submitted it as a request.
   1. Follow-up question: How did the requestor submit that CSR? For example, did they log in as a 'customer' account and submit that to be associated with the account? Did they have their own account? Something separate?
   2. Follow-up question: Do you have the original CSR?
2. This was received and assigned to a validation agent
3. The Applicant then informed the validation agent that they would like to update the domain, from the domain originally in the CSR to the domain in the certificate (bdo-aws-lca.bdousa.com)
4. The Validation Agent updated the request from what was in the CSR to what the applicant requested
   1. Follow-up question: What steps are involved in this? While I can imagine this may be a 'routine' task (since there's no requirement that a CSR include the requested domains), I'm curious to understand what existing controls around this process are.
5. The Validation agent then indicated in the system that they had confirmed, via phone validation, that the Applicant controlled the domain

The issue was that the Validation Agent believed this exercise was within a test environment, and thus did not practice the existing procedures for updating and validating the domain, correct?

This leads to a few follow-up questions:

1. What is the user experience for your Applicants to make requests - that is, do they make requests for 'testing' and 'production' using the same accounts and application scenarios?
2. What is the user experience for your Validation Agents when in 'testing' versus 'production'. That is, do they use a common interface and set of accounts to review requests, and designate which environment should be used then, or do they have separate interfaces and accounts?
3. What artifacts are recorded for when phone validation is performed? Have you ensured these artifacts are consistent for validations that have used this method?
   1. For example, does your validation system show that you made domain lookups to determine the phone number (3.2.2.4.3), your phone/PBX system show your validation agents made calls to those numbers, and the information the validation agent entered is consistent with that?
   2. Have you reviewed all other validations by this agent?
   3. Have you reviewed all other phone validations?
4. Have you examined whether a 'four-eyes' principle (four-ears?) for validations that require humans in the loop, such as 3.2.2.4.3?

While I understand that it's reported as a training issue, understanding the system design helps understand whether or not training issues are likely to happen, and what systemic steps can be taken to mitigate this.

Comment 4

[fcorday]

An employee of Trustwave created a CSR to demonstrate a proof of concept of what a managed service customer would experience, and submitted it as a request.
Follow-up question: How did the requestor submit that CSR? For example, did they log in as a 'customer' account and submit that to be associated with the account? Did they have their own account? Something separate?

The CSR was emailed to our validation team. The validation agent logged into our internal certificate issuance system and accessed our test account named "SSL Test" to which he and others have access. The unfortunate naming of the test customer was one source of the confusion for the validation agent in thinking that this led to a not-publicly-trusted certificate.

    Follow-up question: Do you have the original CSR?

Yes.

-----BEGIN CERTIFICATE REQUEST-----
MIIC2zCCAcMCAQAwgZUxCzAJBgNVBAYTAlVTMREwDwYDVQQIDAhJbGxpbm9pczEQ
MA4GA1UEBwwHQ2hpY2FnbzEbMBkGA1UECgwSVHJ1c3R3YXZlIEhvbGRpbmdzMRsw
GQYDVQQDDBJ3d3d3LnRydXN0d2F2ZS5jb20xJzAlBgkqhkiG9w0BCQEWGHRhY3N1
cHBvcnRAdHJ1c3R3YXZlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKOq2Yi9ChWPianQVbudxurruTbhTsL8BagIuQhNo1Q1W6pLi8VyM24bbycH
fupttSp66kbqYWE7IzzHnmgv6Vnj+Gh/WvRj0EO7q1LFaz9ym/mqdnoxAFuyAuY+
8+4Pwyff6b2shplGozVzl2ICdyNU9Y5WJD5DoWzfgg7UQzL7mtwZAn4wHGe1n57y
nR5/5/GfYoKle9aWzwLbkfnk6QiYwitMEziTEGpdnbe4JCgnjtcQ9VjjeEpmx4Wq
f5BTBPIYauxfIGoFl9WnaJegQpAT+n6wwrrhADZmOJ15U1F9BgE3kgsTAWQR2qRj
2eR3AE2WxzAFRccG3i6jEtmotwUCAwEAAaAAMA0GCSqGSIb3DQEBBQUAA4IBAQCa
Sxe3RaKRikgXH61ISqZjw69Z6vVxxIvWSJ6htDs2E9Pe59zptXLaa0k+DQ1uWoUt
ZrZ35LEReuULQe7ZIGTSuj18CihQmhL/V7xUd3oNOfpOtZg2Fz/uWv1ufjfeJzlj
OSHnR/KChSP47oZanaDT5UDoBWs5/VRdi8y1v/UqREk+HRBEEicANLUChGI+j6XE
mqlBxgZsIMk6puo99K4Vsg8JKwHkDv3cZtlTo/lityVbrE7tYyZeYHzRL/uttBdR
FjRrOkHUJCjP368vI/MCDEu8fJmMWPpldiy0CkK/HTqJFl40N5uarMXgGP8rjcDz
3MKmeSgbXZTo0qMRqVt+
-----END CERTIFICATE REQUEST-----

This was received and assigned to a validation agent
The Applicant then informed the validation agent that they would like to update the domain, from the domain originally in the CSR to the domain in the certificate (bdo-aws-lca.bdousa.com)
The Validation Agent updated the request from what was in the CSR to what the applicant requested
    Follow-up question: What steps are involved in this? While I can imagine this may be a 'routine' task (since there's no requirement that a CSR include the requested domains), I'm curious to understand what existing controls around this process are.

Validation support personnel have the ability to remove or add any of the domains, including the first one, using a separate internal application prior to validation. There are no specific limitations on this capability internally, beyond the fact that they must have access to the application and a user role that allows them to do certificate request validation.

The Validation agent then indicated in the system that they had confirmed, via phone validation, that the Applicant controlled the domain

The issue was that the Validation Agent believed this exercise was within a test environment, and thus did not practice the existing procedures for updating and validating the domain, correct?

Correct.

This leads to a few follow-up questions:

What is the user experience for your Applicants to make requests - that is, do they make requests for 'testing' and 'production' using the same accounts and application scenarios?

Making a request for 'testing' in this sort of scenario is not common. Typically requests for certificates for internal use go through the same channel as external customer requests which is our Control Center Portal. Our IT department has a customer account and they use our external site to submit a CSR and validate their domains. The request shows up in a queue for the validation support personnel to perform additional validation based on the type of certificate.

This request did not go through the IT channel. It was requested directly from a different department via email to our validation team. This department has very limited experience dealing with SSL certificates working with our SSL support team. Based on the validation agent’s belief that the "SSL Test" customer was not issuing "real" certificates, and with his specialized access user role of being able to execute manual validations, he was able to complete the validation and issue the certificate.

What is the user experience for your Validation Agents when in 'testing' versus 'production'. That is, do they use a common interface and set of accounts to review requests, and designate which environment should be used then, or do they have separate interfaces and accounts?

Our Validation Agents do not have access to a testing environment. In those occasional situations where a true test certificate is needed beyond our normal Software Development Lifecycle processes, this is handled by engineering or QA.

What artifacts are recorded for when phone validation is performed? Have you ensured these artifacts are consistent for validations that have used this method?
    For example, does your validation system show that you made domain lookups to determine the phone number (3.2.2.4.3), your phone/PBX system show your validation agents made calls to those numbers, and the information the validation agent entered is consistent with that?
    Have you reviewed all other validations by this agent?
    Have you reviewed all other phone validations?
Have you examined whether a 'four-eyes' principle (four-ears?) for validations that require humans in the loop, such as 3.2.2.4.3?

This is the only instance of using 3.2.2.4.3 that has been executed, which is why it caught our attention. There is no automation around this method. The evidence recorded would be manually entered by the individual performing the validation.

The other validation methods we perform (2, 3, 6, and 7) are executed by the system, and the typical workflow is for requests to be made available for our validation agents only after these steps have been completed by the customer working with the automated systems.

Our validation team uses the 4 eyes principle extensively already. This particular case, it was not used as it was believed to be a test scenario. We are examining further implementations based on this incident report.

Comment 5

[Firefox Bug Husbandry Bot]

Correcting bug type to task.

Comment 6

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.