1409735 - RapidSSL CAA Mis-Issuance: Lookup failure on DNSSEC-signed zone

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Quirin Scheitle]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36

Steps to reproduce:

I requested a certificate from RapidSSL for a test domain.
That test domain is DNSSEC-signed with a valid chain to the root. 
It has a "issue ;" CAA record, however the server is configured to not reply to CAA queries.
This results in a timeout for CAA queries.
As the zone is validly signed, the lookup failure MUST NOT be interpreted as permission to issue. 

This is stated in CAB Ballot 187:
"CAs are permitted to treat a record lookup failure as permission to issue if:
* the failure is outside the CA’s infrastructure;
* the lookup has been retried at least once; and
* the domain’s zone does not have a DNSSEC validation chain to the ICANN root.
"

Certificate: https://crt.sh/?id=228963368
DNS Zone setup: http://dnsviz.net/d/gazebear.org/Wd3l6g/dnssec/

I contacted RapidSSL on October 13 at 11:43 CEST under [e-mail] and [e-mail].

We have ongoing conversation, but not confirmation of mis-issuance yet.

This is part of a larger CAA issuance experiment with more documentation under https://github.com/quirins/caa-test


Actual results:

I was issued a certificate


Expected results:

I should not have been issued a certificate

Comment 1

[Jeremy Rowley]

Here's the incident report from DigiCert:
1.	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2017-10-13 at 11:48 PDT, Quirin Scheitle from Technical University of Munich submitted a certificate problem report to (then) Symantec's technical support team stating that a certificate should not have been issued. Quirin was forcing CAA response supression and had signed the zone with DNSSEC. This bug  was opened on 2018-10-18 at 6:32 PDT.

2.	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2017-10-13 11:48 PDT: Complaint received
2017-10-18 11:44 PDT: Issue confirmed, revocation ordered, analysis started, daily senior management review meetings started
2017-10-19 10:01 PDT: Certificate revoked
2017-10-20 17:19 PDT: Cause determined to be incorrect handling of CAA response from either a TLD or a public suffix. In cases where CAA fetch timed out at the Base Domain, software rules improperly treated a successful empty set fetch from the TLD or a public suffix as permission to issue. Specifications for a patch were written and submitted for review and signoff.
2017-10-22 (various): The same certificate requestor applied for three more certificates from the same domain. Two of these were issued, one from a Thawte test hierarchy and one from a GeoTrust production hierarchy. The third request was queued for manual review and denied. Revocation of the GeoTrust certificate was ordered.
2017-10-XX: Symantec scanned all certificates issued since September 8 and found ### additional certificates that were issued as a result of this bug.
2017-11-1: Symantec's certificate operations transferred to DigiCert
2017-11-3: DigiCert team investigated and decided to a) revalidate any certs that didn't have a complete CAA record and b) apply a patch to correct DNSSec record checking.
2017-11-9: Applied patch to fix CAA record checking
2017-11-13: Identifier issue where the patch did not apply if the service timed-out. This impacted one certificate, which was later reconfirmed as not having DNSSEC present. 

3.	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We installed a patch to stop treating a successful fetch of a CAA empty set at the TLD or PSL entry as permission to issue on 2017-10-26 at 11-9-2017. 

4.	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Four certs were initially identified through a complete database scan as failing to reject the order when DNSSEC was present:

659819b6a372f54974c3a2152ea29790	https://crt.sh/?id=206206566
52797a9df5277a59bc791b9a159e9537	https://crt.sh/?id=222136399
714908fafa3da410261169e3b28ce0b0	https://crt.sh/?id=215460392
633c8ebf67af1b1978f8be587e72166d	https://crt.sh/?id=225108715

All four were revoked. One additional cert was identified post the patch, but was reverified as okay post-issuance.  We didn't revoke the cert as the CAA records stored (at the time of issuance) confirmed the cert was not mis-issued, even if the system didn't properly complete the DNSSEC part of the check. 

5.	The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

659819b6a372f54974c3a2152ea29790	https://crt.sh/?id=206206566
52797a9df5277a59bc791b9a159e9537	https://crt.sh/?id=222136399
714908fafa3da410261169e3b28ce0b0	https://crt.sh/?id=215460392
633c8ebf67af1b1978f8be587e72166d	https://crt.sh/?id=225108715

6.	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The test cases were incomplete and did not adequately test for the case where CAA fetch for the base domain times out and checks either the public suffix or TLD. While the system does properly handle a non-empty CAA record fetch at the TLD in the cases where we are and are not permitted to issue, the testing did not reveal that the system treated an empty set at the TLD as permission to issue. In test cases where we successfully retrieved an empty set at the base domain, our tests obscured the fact that we continued to the suffix or TLD and also treated that successful empty set as permission to issue.

7.	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
We implemented a patch that corrects empty set handling at the PSL and TLD levels while continuing to process positive CAA record result sets. The patch causes certificate requests to display CAA processing as failed in the case that CAA times out at each check required by RFC 6844 except at the TLD or a public suffix unless the CAA at the PSL or TLD states that our brands can issue. When a request fails CAA checking, it is held in pending state for human review.

Comment 2

[Gervase Markham [:gerv]]

That's a great report, Jeremy, thanks. One thing:

> 2017-10-XX: Symantec scanned all certificates issued since September 8 and found ### additional certificates that were issued as a result of this bug.

This doesn't seem to have been correctly filled in :-)

Gerv

Comment 3

[Quirin Scheitle]

This looks like a great incident response and issue resolution, thank you! 
I will occasionally retest with a new domain that is not on the watchlist ;)

Kind regards
Quirin

Comment 4

[Jeremy Rowley]

Oops! The additional certs were zero. There were five total certs found. 

Quirin - feel free to test as much as you want, especially if you wait until after Dec 1. At that time, DigiCert systems will be handling all of the CAA record checking instead of Symantec's legacy systems.  Can't wait for you to find the holes in ours!

Comment 5

[Wayne Thayer]

It appears that all actions have been completed and questions have been answered, so I am marking this resolved.