1531817 - DigiCert: in-addr.arpa Misissuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Cynthia Revström posted the following message to the mozilla.dev.security.policy mailing list:

I have managed to issue a certificate with a FQDN in the SAN that I do
not have control of via Digicert.

The precert is here: https://crt.sh/?id=1231411316

SHA256: 651B68C520492A44A5E99A1D6C99099573E8B53DEDBC69166F60685863B390D1

I have notified Digicert who responded back with a generic response
followed by the certificate being revoked through OCSP. However I
believe that this should be wider investigated, since this cert was
issued by me adding 69.168.110.79.in-addr.arpa to my SAN, a DNS area
that I do control though reverse DNS.

When I verified 5.168.110.79.in-addr.arpa (same subdomain), I noticed
that the whole of in-addr.arpa became validated on my account, instead
of just my small section of it (168.110.79.in-addr.arpa at best).

To test if digicert had just in fact mis-validated a FQDN, I tested with
the reverse DNS address of 192.168.1.1, and it worked and Digicert
issued me a certificate with 1.1.168.192.in-addr.arpa on it.

The discussion can be found here: https://groups.google.com/d/msg/mozilla.dev.security.policy/JFwqZx7RLL0/My83uQA5AAAJ

Comment 1

[Wayne Thayer]

Jeremy Rowley posted the following response:

We've figured out what happened with your certificate but are still looking at whether other certificates were issued using the same process. I don't have enough information to file an incident report yet, but I wanted to give you the update I promised earlier.

Basically, here's what happened:

1. A validation agent took an email address provided during a chat session with you and set that email as a WHOIS admin email address. This email qualified as a constructed email (admin@domain)
2. The system marked the WHOIS as unavailable for automated parsing (generally, this happens if we are being throttled or the WHOIS info is behind a CAPTCHA), which allows a validation agent to manually upload a WHOIS document
3. The WHOIS document uploaded was a screen capture of WHOIS information that did not match the domain being requested but was close enough that the mis-match went unnoticed.
4. The validation agent specified the approval scope as id-addr.arpa which is normal for a domain approved by the admin listed in WHOIS. As a constructed email, the approval scope should have been limited to the scope set by the constructed address.
5. The validation agent specified that the email address listed in WHOIS matched the email address provided by you (admin@domain) and sent the email to authorize the legit cert
   6 . When the validation completed successfully, the validation authorized your account for all certificates with the in-addr.arpa domain. This was because the scope was improperly set based on the validation agent's improper specification of the source of the email address.
6. During the review, no one noticed that the WHOIS document did not match the verification email nor did anyone notice that the email used for verification was actually a constructed email instead of the WHOIS admin email.
7. The mis-issued certificate essentially "piggy-backed" on the previous verification because of the erroneous approval scope set by the validation staff.

Make sense?

We shut down manual WHOIS verification while we figure out how to improve the process and eliminate validation mistakes like this. We'll post another update after we figure out how to improve the process and after the data review is complete.

Comment 2

[Jeremy Rowley]

Incident report:Incident Report – Mozilla Policy Violation

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Cynthia Revstrom directly emailed DigiCert’s CTO and principal product manager on 26 Feb 2019 at 5:20am MT. The PM acknowledged at 7:52am MT. The [e-mail] alias was not used. Cynthia then posted to MDSP at 11:02am MT.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

(All times US MST, GMT-7)

26 Feb 2019:
5:20am: Notification from customer sent directly to two DigiCert employees
7:47am: Investigation started
7:52am: Response to customer confirming receipt of problem report
7:52am: Determined that .arpa is an allowed TLD and that the Domain Control Validation (DCV) email was sent to [e-mail] due to a validation agent manually uploading
a WHOIS document and manually configuring the DCV email recipient
8:07am: Revoked the DCV for in-addr.arpa in Cynthia’s account
8:15am: Determined that the Authorization Domain Name scope was set to in-addr.arpa when the DCV email was sent.
9:48am: Certificates https://crt.sh/?id=1231235765 and https://crt.sh/?id=1231411316 revoked
11:02am: Cynthia posts to MDSP, also documenting our revocation

27 Feb 2019:
6:31pm: Manual upload of WHOIS documents disabled for non-managers

We also performed system-wide scan for any certs where a manual WHOIS validation was used and where the domain scope did not match the email address used. We found 3,000 instances. Upon review, none of the changes violated the BRs.

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We stopped issuing certificates within in-addr.arpa on 26 Feb 2019 at 8:26am MT when we set the privileges required to validate any manual change in validation scope to the EVP of Validation and his direct reports.

4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

See #5.

5.The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1231235765 – Initial cert issued - 25 Feb 2019 9:04am MST
https://crt.sh/?id=1231411316 – Second cert issued - 25 Feb 2019 9:59am MST

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

When domain contacts cannot be automatically retrieved and parsed from WHOIS for use in an automated DCV under 3.2.2.4.2, we allowed (until 27 Feb 2019) validation agents to manually upload a WHOIS document. When this feature is used, validation agents were able to specify a domain contact email address based on the content of the manual retrieval. Our process relies on the knowledge and training of our validation agents because any document can be uploaded. We do audit the staff during our regular periodic self-audits to ensure a domain validation document is uploaded and the email address matches the email listed in the domain document.

In this situation the validation agent uploaded a WHOIS document that did not represent control over in-addr.arpa. They then manually constructed a DCV email address of [e-mail]. The uploaded document was a screenshot of WHOIS but not the WHOIS for in-addr.arpa, which is not available for automated parsing.

When a validation agent specifies a domain, the software uses a domain intelligence ruleset to calculate Authorization Domain Names (ADN). Because we had no explicit rule about .arpa, the default rule of TLD plus one left label was used, resulting in an ADN of in-addr.arpa. The agents can narrow this scope. This did not happen with the certs in question.

When the agent prepared to send the DCV to the [e-mail] recipient, they would normally ensure the scope matches the right hand side of the email. Constructed emails using the domain are set to match the right-hand side of the email address automatically. Because this was not a constructed email (despite looking like one) and was entered as a WHOIS recored based email, [e-mail] was granted the right to approve all certificates ordered through their account for in-addr.arpa.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

At 8:26am MT on 26 Feb 2019, we elevated the privilege to process domain validations for the .arpa TLD to EVP and direct reports only. On 27 February 2019, we disabled the features that allow validation agents to manually upload WHOIS documents or specify the domain contact email address of a 3.2.2.4.2 DCV.

Lack of training on the proper use of a feature designed to resolve automated WHOIS retrieval problems caused this isolated case of a validation agent mis-using the upload process for a domain with WHOIS information that is not relevant to the applicant. Manual WHOIS upload is disabled for non-managerial staff until we can write additional software controls that correlate the uploaded document to the domain and parse out the domain contact email addresses. Feature training would be part of our mitigation if we intended to re-enable this feature for all validation staff, but without the technical controls required, we will not. Remedial training to share how this was mis-used and why it is a problem has been addressed.

Comment 3

[Ryan Sleevi]

Wayne: I surprisingly don't have any further comments. DigiCert's doing much better in their incident reports than where they were at, and certainly where other CAs are at. I'm not sure if you had further questions regarding this incident.

Comment 4

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.