1409766 - Asseco DS / Certum: CAA Mis-Issuance on CNAME pointing directly to restrictive CAA record

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Quirin Scheitle]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36

Steps to reproduce:

I set up a test domain, whose www label is a CNAME.
This CNAME points *directly* to a different domain with a "issue ;" CAA record. 
(Directly means there is no parent zone climbing at the CNAME target necessary).

www.gazebear.online -> CNAME -> gazebear.net 
gazebear.net CAA record: "issue ;"

Certum issued a certificate for the www.gazebear.online, which it should not have.
While the CN only lists gazebear.online, the certificate's SAN also includes www.gazebear.online.

Certificate: https://crt.sh/?id=230122233
Zone: http://dnsviz.net/d/www.gazebear.online/WeR_8w/dnssec/

I communicated this issue to Certum on Oct 16, 14:16 CEST and have been in communication with them since then. However, a root cause could not be identified yet.

This is part of a larger CAA issuance experiment with more documentation under https://github.com/quirins/caa-test

Comment 1

[Gervase Markham [:gerv]]

Arkadiusz: do you have an update here/ It's been a month since this bug was filed.

Gerv

Comment 2

[Arkadiusz Ławniczak]

Yes. We confirm that some problems have been observed during the last months when processing CAA checking for subdomain names.
 

Our plan is as follows:
1. All issued certificates will be audited in the next weeks relating to CAA checking for all names within each SAN per certificate.
2. Some part of CAA verification proccess in CERTUM remains semi-manually as far.
3. Our automated system for verifying CAA records will be improved by the Q2 2018.
4. By that time, in order to ensure that CAA checking is fully compliant with RFC 6844, CAA verification will be performed for each issued certificate also after it has been issued but no later than in the day of issuance.
5. Any case of non-compliance where the issuances were	prevented by a CAA record will be provide to the CAB Forum according to 3.2.2.8 of Baseline Requirements.

Comment 3

[Gervase Markham [:gerv]]

Arkadiusz: from what you say in point 4, it sounds like you are doing CAA checking after issuance rather than before. Is that correct?

You say that some parts remain manual. Does that mean that humans are attempting to execute the CAA algorithm in their brains, doing the required DNS queries manually? That sounds to me like it would be very error-prone.

Gerv

Comment 4

[Arkadiusz Ławniczak]

No. Of course we check CAA before issuance. Point 4 says that in the face of problems with verification of which we are aware, and which we are not able to resolve within the verification track untill to update our system as a whole (what will take place by the end of Q2 2018 or earlier), we decided to make additionall CAA checking also after issuance as to exclude any potential doubts regarding to issued certificates.

Comment 5

[Wayne Thayer]

Arkadiusz: Please provide a full incident report as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Also, please provide an update on the system improvements for verifying CAA records.

Comment 6

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 7

[Wayne Thayer]

Arkadiusz: this is awaiting your response. Also, have the automated CAA verification improvements described in comment #2 been deployed?

Comment 8

[Wojciech Trapczyński]

Hi Wayne,

the automated CAA verification described in comment #2 has been deployed. 

We were operating in accordance with Arkadiusz description till this time.

Wojciech Trapczyński

Comment 9

[Wayne Thayer]

Wojciech: thank you for this information. Now we are just waiting for the incident report that I requested in comment #5. Please respond with this information as soon as possible.

Comment 10

[Wojciech Trapczyński]

I will send it shortly.

Comment 11

[Wojciech Trapczyński]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

October 16, 2017 - Certum received the message from Quirin Scheitle.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Oct 12, 2017 - The certificate was issued.
Oct 16, 2017 - Certum received the message with bug description from  Quirin Scheitle.
Oct 18, 2017 - Certum found the reason for this bug.
Oct 18, 2017 - Quirin Scheitle reported bug on Mozilla Bugzilla.
Oct 24, 2017 - Certum deployed the validation procedure (with dedicated application) for all certificates which might be misissued.
Nov  7, 2017 - Certum revoked the certificate.
Sep 11, 2018 - Certum deployed fixed CAA validation module on production.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

For OV and EV certificates on October 24, 2017.
For DV certificates on September 11, 2018.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The total number of certificates with the problem is: 1.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=230122233

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The misissue was caused by improper CAA validation for domains with www prefix and CNAME record. For such names we performed CAA validation but it always gave a positive result. 

For example:

www.gazebear.online -> CNAME -> gazebear.net 
gazebear.net CAA record: "issue ;"

gives positive result and the certificate was issued, but for

gazebear.online -> CNAME -> gazebear.net 
gazebear.net CAA record: "issue ;"

gives the negative result and the certificate was not issued.

The error in validation module was not found by our tests suite because we did not have develope scenario for such case.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We fixed CAA validation module to prevent such issuance. Till this time, we have made actions described below:

- We verified and fixed a flow diagram for CAA validation considering all necessary steps described in RFC 6844. On this basis, we developed a standalone application to help vetting team in performing CAA validation for domains with www prefix and CNAME record. For OV and EV certificates it was performed before the issuance. For DV certificates it was performed after the issuance, but no later than 24 hours since issuance. Any misissued certificate was not found.  
- Additionally, we have checked all issued certificates since Sep 8, 2017 and we did not find any other certificate which was issued without proper CAA validation (except that one reported in this bug).
- We have verified tests suite. We have added the test case for scenario with www prefix and CNAME record.

Comment 12

[Wayne Thayer]

Wojciech: Thank you for providing the incident report. Your statements seem to contradict what Arkadiusz wrote in comment #4. Is it correct that Certum was not performing CAA checking on DV certificates before issuance prior to September 11, 2018? If so, is your auditor aware of this non-conformity with the Baseline Requirements?

Comment 13

[Wojciech Trapczyński]

No, it is not correct. We started performing CAA validation before issuance for DV, OV and EV certificates since September 8 2017, as required by Baseline Requirements.

The date September 11, 2018 applies only for particular DV certificates with www prefix and CNAME record. For such names there were a risk of misissuance and we needed to perform additional CAA validation also after issuance.

Comment 14

[Wayne Thayer]

Thank you, now I understand. Were any mitigations in place to prevent misissuance of a certificate containing a domain with a www prefix and a CNAME record, or was the only mitigation for this vulnerability the post-issuance check?

Also, can you explain why it took almost a year to fix this vulnerability?

Comment 15

[Wojciech Trapczyński]

For the OV and EV we have checked it before issuance and for DV the only mitigation was a post-issuance check.

A long time to deploy this fix on production resulted from the architecture of our application. We treated www prefix in different way from others subdomains. Therefore, we needed to rebuild a significant part of our software. We were ready to deploy it on production at the end of Q2 2018 (in June). Unfortunately, during tests on the model enviroment we detected that the implemented changes had a negative impact on the performance of the entire CAA validation. For this reason, deploy fix on production has been delayed.

Comment 16

[Ryan Sleevi]

Can you please supply the documentary evidence you were required to collect consistent with the CA/Browser Forum's "NETWORK AND CERTIFICATE SYSTEM SECURITY REQUIREMENTS" 4.f? This is reflected in Principle 4, Criterion 4.6 of "WEBTRUST PRINCIPLES AND CRITERIA FOR CERTIFICATION AUTHORITIES – SSL BASELINE WITH NETWORK SECURITY" (v2.3) or REQ-7.7-09 of ETSI EN 319 401 (v2.2.1)

Comment 17

[Wojciech Trapczyński]

We have the documentary evidence.

We have created the ticket for this bug in our internal bug tracking system. The ticket has been created on the same day the error was reported on Bugzilla. Then we have made an analysis and we have calculated a risk. We have estimated that a probability of an error occurring is low. On this basis, we scheduled to implement and deploy the fix in Q2 2018 (then it was postponed until Q3 2018 due to detected problems on test enviroment).

We were aware that for SSL DV certificates the misissue may occur till we deploy the fix and for that reason we were performing the post-issuance checks to detect it in a quick manner.

Comment 18

[Wayne Thayer]

(In reply to Wojciech Trapczyński from comment #17)
> We have the documentary evidence.
> 
> We have created the ticket for this bug in our internal bug tracking system.
> The ticket has been created on the same day the error was reported on
> Bugzilla. Then we have made an analysis and we have calculated a risk. We
> have estimated that a probability of an error occurring is low. On this
> basis, we scheduled to implement and deploy the fix in Q2 2018 (then it was
> postponed until Q3 2018 due to detected problems on test enviroment).
> 
We are now well in to Q4 2018 and this message implies that the fix has still not been deployed. If so, when will it be deployed?

> We were aware that for SSL DV certificates the misissue may occur till we
> deploy the fix and for that reason we were performing the post-issuance
> checks to detect it in a quick manner.

Are you sampling 100% of issued certificates? Has any misissuance been discovered?

Comment 19

[Wojciech Trapczyński]

We deployed fix on 2018-09-11 as I described in the Incident Report and this day is the last one when the misissuances for SSL DV certificates may have occurred.

Yes, we review logs for all certificates issued between 2017-09-08-2018-09-11 and as I described in the Incident Report we did not find any additional misissuances.

Comment 20

[Ryan Sleevi]

Wayne: I think this goes back to you for review and final decision.

Comment 21

[Wayne Thayer]

It appears that remediation is complete. Resolving.