Closed Bug 1483715 Opened 6 years ago Closed 6 years ago

DigiCert: improper use of domain validation method

Tracking

(Not tracked)

Status:

RESOLVED FIXED

People

(Reporter: jeremy.rowley, Assigned: jeremy.rowley)

Details

(Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance])

Jeremy Rowley

Assignee

Description

•

6 years ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 Steps to reproduce: 1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. On 2018/08/07 at 17:00 UTC, a customer submitted a request for information about our validation process for the verification of four of their domains. Upon investigation, we found that the four domains were not properly validated using a post-Aug 1 domain validation method. When attempting to revalidated the domains prior to August 1, the random value was sent to an address other than the WHOIS contact. This launched a broader investigation into our overall revalidation efforts. This investigation is ongoing. 2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. From approximately February through April 2018, DigiCert permitted some legacy Symantec customers to use Method 1 to validate their domains. Use of the method was subject to manager approval and reserved only for those companies that had urgent replacement deadlines that could not be met with an alternative validation method. Under this process, prior to approval, the validation staff was required to match the WHOIS company information and obtain approval using the WHOIS email address. Around April, this process was modified to include a BR-compliant Random Value that the validation staff sent using the WHOIS contact information. Use of the random value indicated acceptance. Adding the random value effectively transformed the validation from Method 1 to Method 2. The email could include multiple domains with the understanding that the WHOIS contact information had to match each domain listed. We believe that in some cases either the validation staff failed to match the WHOIS contact information for each domain listed, approving the certificate solely based on the existing verified registrant info, or the system did not check whether the WHOIS contact information matched the email address used in the original confirmation. On Aug 1, 2018, Ballot 218 took effect, deprecating Method 1. On, August 7, 2018, a customer requested the audit trail of a certificate issued using our new process. Upon review, validation management discovered the validation was improper because the previously verified email contact information did not match the WHOIS contact information. This discovery created an escalation up to management. On August 13, 2018, we stopped all issuance based on the process that converted Method 1 validations to Method 2 validations. We’re currently investigating and will post an update when we know the number of certificates and more about what went wrong. For now, we know the number of impacted certificates is just under 2,500. We should have a clearer picture shortly, after we have conducted a manual review of all 2,500 certificates. 3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. On August 13, although most of these validations were likely properly completed, we stopped issuance using information converted from Method 1 until completing a more thorough investigation. 4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. Approximately 2,500 certificates are under review for validation issues. We wanted to get the incident report out quickly as an FYI while our investigation continues. We’ll update this section in a final report. 5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. Still under review. We will upload them to the bug once we have a complete list. Because the error was human, we are reviewing each validation to determine whether Method 2 was correctly used. Once we complete our review, we’ll post a Bugzilla attachment with the links for revoked certificates. 6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. Preliminarily, we believe that the revalidation process that relied on previously verified information failed because there were not technical controls in place to ensure information integrity and because that the validation staff received insufficient training on the process of uploading WHOIS information., This resulted in validation staff adding some domains to the approval email where the WHOIS contact information didn’t always match all the other domains listed. When we performed revalidation of domains in preparation for Ballot 218’s effective date, we considered the existing domain information to be correct and reusable until the normal expiration date. However, this inadvertently blended poor quality validations due to the failure to check WHOIS contact information with properly completed Method 2 validations, allowing certs to issue post Aug 1 without proper domain validation. 7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. We stopped all issuance relying on impacted domains until the domain control is revalidated. We are also investigating each impacted domain’s validation to determine whether the email was sent to the appropriate WHOIS contact. Any certs issued based on an improper validation will be revoked or revalidated using an approved method. We will update this issue and post to the Mozilla list when we have a precise number of impacted certs. We’re still considering what additional action to take and will post an update when we figure out more about what went wrong.

Wayne Thayer

Updated

•

6 years ago

Assignee: wthayer → jeremy.rowley

Summary: Improper domain validation → DigiCert: improper domain validation

Whiteboard: [ca-compliance]

Jeremy Rowley

Assignee

Comment 1

•

6 years ago

Ryan Sleevi

Comment 2

•

6 years ago

(In reply to Jeremy Rowley from comment #0) > On 2018/08/07 at 17:00 UTC, a customer submitted a request for information > about our validation process for the verification of four of their domains. > Upon investigation, What triggered the investigation? When did the investigation begin? Helping understand this can help make the difference between "We got lucky" and "Systemic controls such as X, Y, Z noticed this, and all CAs can benefit from implementing such controls" > 2. A timeline of the actions your CA took in response. A timeline is a > date-and-time-stamped sequence of all relevant events. This may include > events before the incident was reported, such as when a particular > requirement became applicable, or a document changed, or a bug was > introduced, or an audit was done. > From approximately February through April 2018, DigiCert permitted some > legacy Symantec customers to use Method 1 to validate their domains. Use of > the method was subject to manager approval and reserved only for those > companies that had urgent replacement deadlines that could not be met with > an alternative validation method. Under this process, prior to approval, the > validation staff was required to match the WHOIS company information and > obtain approval using the WHOIS email address. Can you expand a bit more to discuss what these controls look like. "Prior to April 2018, domain validation was performed using the following steps: 1) A CSR is submitted containing domains 2) For each domain, our systems perform a lookup for the WHOIS information, displaying it to the validation staff. 3) The validation staff copies or transcribes this text into creating an email order using the WHOIS email to obtain authorization to issue on the basis of the Company Match 4) A random value is ..." Or something to that effect? Help us understand how the existing system of controls were designed. > Around April, "Around April" is very precisely date-and-time-stamped :) > this process was modified to include a BR-compliant Random > Value that the validation staff sent using the WHOIS contact information. > Use of the random value indicated acceptance. Adding the random value > effectively transformed the validation from Method 1 to Method 2. The email > could include multiple domains with the understanding that the WHOIS contact > information had to match each domain listed. Here again, with the above details about how the system 'used' to work, it can be clearer how the system 'does' work. > We believe that in some cases either the validation staff failed to match > the WHOIS contact information for each domain listed, approving the > certificate solely based on the existing verified registrant info, or the > system did not check whether the WHOIS contact information matched the email > address used in the original confirmation. Here, understanding the exact processes above help understand why it failed. "We believe" doesn't inspire strong confidence, because it suggests that processes or controls are missing from the workflow, or that the binding between requests and domains is a manual process that permits human error, etc. This is about describing the system architecture to help identify where things went right - and wrong. > On, August 7, 2018, a customer requested the audit trail of a certificate > issued using our new process. Is this typical? Was this customer reporting unauthorized issuance? > Upon review, validation management discovered > the validation was improper because the previously verified email contact > information did not match the WHOIS contact information. This discovery > created an escalation up to management. When was it escalated? When was it acknowledged? -and-timestamp helps understand what the processes and response times are. Postmortems help identify where things go right as much as they identify where things went wrong - it's often the things going right that prevent them from being far worse than they could have been, and serve as valuable tools to understand how to build and improve safe systems, which is part of the goal of these processes :) > We’re currently investigating and will post an update when we know the > number of certificates and more about what went wrong. For now, we know the > number of impacted certificates is just under 2,500. We should have a > clearer picture shortly, after we have conducted a manual review of all > 2,500 certificates. What does your manual review entail? By understanding all the above facts, and how the manual review is being done, we can better have confidence that it's "just" those 2,500 certificates. While it's very good to be responsive and identify the issue, it's also important to be thorough, and by sharing information, we build confidence in the thoroughness.

Jeremy Rowley

Assignee

Comment 3

•

6 years ago

> What triggered the investigation? When did the investigation begin? Helping understand this can help make the difference between > "We got lucky" and "Systemic controls such as X, Y, Z noticed this, and all CAs can benefit from implementing such controls" Investigation was triggered by a customer requesting information about the validation process with one of their certs. We started investigating on 08/08/2018 (the day after the customer submitted the request). The one day delay was because we were getting more info on what the customer was actually asking. The controls didn't pick up the issue because everything looked right from the document-side. > Can you expand a bit more to discuss what these controls look like. > > "Prior to April 2018, domain validation was performed using the following steps: > 1) A CSR is submitted containing domains > 2) For each domain, our systems perform a lookup for the WHOIS information, displaying it to the validation staff. > 3) The validation staff copies or transcribes this text into creating an email order using the WHOIS email to obtain authorization to issue on the basis of the Company Match > 4) A random value is ..." > > Or something to that effect? Help us understand how the existing system of controls were designed. Sure. Note this is the system for legacy Symantec orders which is slightly different than DigiCert certs. That's not intended to be an excuse, but it means this explanation doesn't apply to certs issued through legacy DigiCert systems. We're working on merging the two, but the progress is slow. The system in question used the following process: 1) Customer submits a domain verification request through their Symantec portal 2) System kicks off email to BR-approved email address 3) If WHOIS is not parse-able (some ccTLDs) or requires captcha, validation staff is alerted to pending (assuming customer does not use a DNS or file based method) 4) Validation staff accesses WHOIS and screenshots information 5) Validation staff uploads WHOIS screenshot and records WHOIS email addresses 6) Validation staff can consolidate orders with the same WHOIS email into a single approval email 7) Validation staff emails WHOIS contact to verify the approval. Under method 1, the validation staff confirmed the org information in WHOIS. A random value was not included (since validation was method 1) Added/replaced in April: 7) System generates a random value (per BRs) 8) System kicks off email to the specified WHOIS address(with link to use random value) 9) Customer clicks link, verifies approval, and gets cert What happened is two fold: a) Some validation staff did not use the WHOIS contact to obtain approval. Instead they did a "real" method 1 and confirmed the org information in WHOIS matched what was verified during the OV/EV process. However, because the system required a WHOIS email address, an email address for the customer was specified that did not necessarily conform to what WHOIS showed. b) When we converted the handful of customers verified using method 1 to method 2, we resent the confirmation email with a random value to the specified WHOIS address. Because the WHOIS information was not accurately recorded, the validation email went to an inappropriate email address. > "Around April" is very precisely date-and-time-stamped :) Yeah. I couldn't find the exact date we rolled the system random value when I wrote the original disclosure. Turns out the actual dates as March 1, 2018. On that same date, we restricted second approval to management. Previously second approval permitted any two staff to confirm the email address recorded as accurate. This lead to some interesting email addresses approved by validation staff. We restricted this to approved roles and expanded it as people received training. > Is this typical? Was this customer reporting unauthorized issuance? Customer requests about validation are not typical. The customer wasn't reporting unauthorized issuance. The customer approved the order through their account but never approved the domain verification and was wondering what process was in place. The request was more curiosity on the process than a complaint about mis-issuance. > Here again, with the above details about how the system 'used' to work, it can be clearer how the system 'does' work. I added details above. Let me know if you want more info about any part of that. > Here, understanding the exact processes above help understand why it failed. "We believe" doesn't inspire strong confidence, > because it suggests that processes or controls are missing from the workflow, or that the binding between requests and domains > is a manual process that permits human error, etc. > > This is about describing the system architecture to help identify where things went right - and wrong. Right. I said "we believe" because we were still investigating the processes. I have a lot more info now that I can share. The architecture prior to March 1 was poor. We rolled out a significant update on March 1 that locked down permissions and automated the process more. > When was it escalated? When was it acknowledged? -and-timestamp helps understand what the processes and response times are. I escalated it to Dev on Aug 13. Prior to that, we were looking through the audit logs of the system to see what we could find out. I can look at my email to find the exact time, but it was in the afternoon. Monday morning was when I was convinced there was a problem. > Postmortems help identify where things go right as much as they identify where things went wrong - it's often the things going > right that prevent them from being far worse than they could have been, and serve as valuable tools to understand how to build > and improve safe systems, which is part of the goal of these processes :) Agreed. Happy to answer any questions you have. > What does your manual review entail? By understanding all the above facts, and how the manual review is being done, we can better have confidence that it's "just" those 2,500 certificates. > While it's very good to be responsive and identify the issue, it's also important to be thorough, and by sharing information, > we build confidence in the thoroughness. Yeah - we were still in investigation phase when I posted. Sorry about the lack of details there. The manual review process is going through every validation completed using this system and looking at the documentation to see what was uploaded by the validation staff. This is an exception system so it only applies where the normal, automated WHOIS failed to send an email with a random value (which is why there aren't a billion certs) and where no other method was usable by the customer. The manual review involves someone trusted (generally an internal auditor) opening up each record, looking at the email address used and ensuring it matched what was in WHOIS. If the email address didn't match, the cert was considered invalid. The certs listed above are the ones where the WHOIS email didn't match the email where the approval was sent.

Dimitris Tsabiras

Comment 4

•

6 years ago

Hi Jeremy, > On August 13, 2018, we stopped all issuance based on the process that > converted Method 1 validations to Method 2 validations. in your certificate list you seem to have included certificates issued on or after 13/8/2018. Is this list correct or did some certificates slipped by the cutoff date? for example https://crt.sh/?id=649151307 https://crt.sh/?id=646734937 https://crt.sh/?id=647142081

Flags: needinfo?(jeremy.rowley)

Jeremy Rowley

Assignee

Comment 5

•

6 years ago

Sorry, we stopped permitting that method for additional validations. Existing validations were not invalidated until we went through each one. These were all certs where the validation completed prior to the cutoff date but didn't issue. We've since invalidated all the validations that lacked random values.

Flags: needinfo?(jeremy.rowley)

Wayne Thayer

Comment 6

•

6 years ago

Jeremy: please provide periodic updates on remediation status and a target completion date if possible. Also, I'm interested in learning what additional actions DigiCert has decided to take, especially in respect to better detection and prevention of this type of issue in the future.

Jeremy Rowley

Assignee

Comment 7

•

6 years ago

Sure thing. We're currently working on the revocation/recheck. I'll have an update on that next week. I'll post what updates we're working on for better detection and prevention at the same time

Jeremy Rowley

Assignee

Comment 8

•

6 years ago

Sorry for the delay on this. Everything was revalidated and approved using a BR-compliant method. As suspected, exactly zero certs were revoked. For better prevention, we're implementing tools that will eliminate some of the manual process that went into the system where this was used. It's mostly already there because Method 1 and Method 5 are no longer allowed and because GDPR eliminated a lot of WHOIS records. However, we want to improve that system to where the process is more automated. Still designing some of this, but generally the validation staff will be more restricted on what WHOIS information can be input. For detection, we're leaning towards using machine learning to better identify when a document is a WHOIS-like document. This way we can detect when a document is not an actual screen shot of WHOIS but something else.

Wayne Thayer

Updated

•

6 years ago

Status: UNCONFIRMED → RESOLVED

Closed: 6 years ago

Resolution: --- → FIXED

Nobody; OK to take it and work on it

Updated

•

2 years ago

Product: NSS → CA Program

David Lawrence [:dkl]

Updated

•

2 years ago

Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance] [ev-misissuance]

Ben Wilson

Updated

•

4 months ago

Summary: DigiCert: improper domain validation → DigiCert: improper use of domain validation method

You need to log in before you can comment on or make changes to this bug.

Bugzilla

Quick Search

DigiCert: improper use of domain validation method

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

People

(Reporter: jeremy.rowley, Assigned: jeremy.rowley)

References

Details

(Whiteboard: [ca-compliance] [ov-misissuance] [ev-misissuance])

Crash Data

Security

(public)

User Story

Description

Updated

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Updated

Updated

Updated

Updated