1398545 - Comodo: CAA Misissuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jonathan Rudenberg]

I just reported the following to [e-mail]:

Comodo has issued the following certificate in violation of the Baseline Requirements CAA checking requirement:

https://crt.sh/?sha256=c102a8110a489520320ed159016226a2994131e1caa4608b665a9e3bd43970fe

The certificate contains the dnsNames caatest.titanous.com and www.caatest.titanous.com. At the time of domain validation and certificate issuance, the effective CAA record set for these domains did not include any Comodo domains:

titanous.com.	300	IN	CAA	0 iodef "mailto:[e-mail]"
titanous.com.	300	IN	CAA	128 issue "letsencrypt.org"
titanous.com.	300	IN	CAA	128 issuewild “;"

There are no CAA records for caatest.titanous.com or www.caatest.titanous.com. According to RFC 6844, the only CA allowed to issue for caatest.titanous.com and www.caatest.titanous.com is letsencrypt.org.

Jonathan

Comment 1

[Andrew]

Interestingly that 4 new root certs of SSL.com now in public discussion for adding - https://bugzilla.mozilla.org/show_bug.cgi?id=1277336

Comment 2

[Quirin Scheitle]

I can confirm this issue. I used 4 test domains with varying restrictive CAA settings and Comodo issued for all 4 domains without checking CAA records (no CAA queries at NS between application and issuance). Details under https://github.com/quirins/caa-test.

My ticket ID with comodo support is #RNZ-386-41223.

Comment 3

[Rob Stradling]

Jonathan, thanks for testing our (non-)compliance and alerting us.  We've taken immediate steps to fix the underlying problem.  We'll follow-up with an incident report ASAP.

Quirin, I haven't seen that ticket but I did see your tweet.  Thanks.

CC'ing Hanno, who also checked and reported our CAA (non-)compliance to our abuse contact.  Thanks Hanno.

Comment 4

[Rob Stradling]

INCIDENT REPORT

We received two Problem Reports - from Hanno Böck on 9th September at 20:10 UTC, and from Jonathan Rudenberg on 10th September at 00:08 UTC - each of which reported that we had misissued a certificate contrary to a published CAA RRset.
Jonathan reported this problem at https://bugzilla.mozilla.org/show_bug.cgi?id=1398545, and in https://bugzilla.mozilla.org/show_bug.cgi?id=1398545#c2 Quirin Scheitle provided a further misissuance report.

TRIAGING
Some Comodo staff saw these reports late on Friday 9th and began to discuss them over the weekend, but they were unable to confirm their accuracy.  Indeed, the reports appeared to them to be erroneous, because the logs at their disposal showed that the relevant CAA checks had been performed but the RRsets were empty.  Therefore, the only action taken at that point was to escalate the reports to the original developer of our CAA checking code to look at first thing Monday morning.

BACKGROUND
As you'd expect from the authors of RFC6844, we were an early adopter, deploying our initial CAA checking implementation 2.5 years ago.  It executes `dig CAA +dnssec +sigchase +trusted-key=dnssec_trusted.keys` to perform the DNS queries.  We chose this approach after concluding that, at that time, it was the least worst option available to us for checking DNSSEC signatures.  We deployed a specific version of BIND (9.10.1-P2) because testing had shown that `dig` in the next release of BIND would crash when trying to do DNSSEC validation.

WHAT WENT WRONG
Our ops team upgraded the servers that our CAA checking code was running on.  This included a very-long-awaited transition from a 32-bit to 64-bit OS.  Rather than recompile 9.10.1-P2 for 64-bit, our ops engineers upgraded BIND to 9.10.5-P1.
Yesterday morning (Monday 11th), when investigating the Problem Reports, the original developer discovered that as a result of that BIND upgrade all of our calls to `dig` were returning the following response:

`Invalid option: +sigchase
Usage:  dig [@global-server] [domain] [q-type] [q-class] {q-opt}
            {global-d-opt} host [@local-server] {local-d-opt}
            [ host [@local-server] {local-d-opt} [...]]

Use "dig -h" (or "dig -h | more") for complete list of options`

Unfortunately, this `dig` response was being interpreted by our CAA checking code as a CAA response that contained: no "issue" property, no "issuewild" property, no unrecognized critical properties, etc.

This problem had gone undetected due to a combination of reasons: the developer did not ask for BIND to be upgraded and so did not expect any behaviour to have changed; the ops engineers did not realize that upgrading BIND might cause a problem; there wasn't an automated test that would've detected this problem and raised an alarm; CAA RRsets are still fairly uncommon, so nobody noticed that we'd dropped from finding hardly any RRsets to finding zero RRsets; our validation staff only see the results of our CAA processing rather than the complete output from `dig`.

ACTION TAKEN TO ADDRESS THE PROBLEM
Upon discovery of the failing `dig` calls, we immediately downgraded to BIND 9.10.1-P2 and verified that our CAA checks were then working correctly.  We also purged our local cache (of recent `dig` responses) to ensure that the misissuance vector was completely closed.

PROBLEM CERTIFICATES
The following certificates have all been revoked:
Reported by Hanno:
https://crt.sh/?id=207082245
Reported by Jonathan:
https://crt.sh/?id=207224651
Reported by Quirin:
https://crt.sh/?id=208456003
https://crt.sh/?id=208486480
https://crt.sh/?id=208486489
https://crt.sh/?id=208486485
https://crt.sh/?id=208486495

NEW CAA CHECKING IMPLEMENTATION
Our initial CAA checking implementation, while functional, was not designed with our current certificate issuance volumes in mind.  Consequently, we had been working on a new, much more scalable CAA checking implementation, written in Go.  We had expected to deploy this new implementation during Q2 2017, but work on this project was paused due to the uncertainties of CNAME processing that have now been resolved at IETF (see https://www.rfc-editor.org/errata/eid5065) and that will hopefully soon also be resolved at CABForum (see https://cabforum.org/pipermail/public/2017-August/011972.html).

DEPLOYING OUR NEW CAA CHECKING IMPLEMENTATION
Having fixed our `dig` calls we found that our system was struggling to process the queue of CAA checks quickly enough, and so we accelerated our plans to deploy our new CAA checking implementation.  This morning (Tuesday 12th) we verified that our new implementation does a reasonable job when faced with the test cases at https://caatestsuite.com/, and we deployed it.

VERIFYING OUR NEW CAA CHECKING IMPLEMENTATION
We are taking immediate steps to engage the services of an external security consultant to independently assess our new CAA checking implementation and to work with us to ensure that it behaves correctly.

ACKNOWLEDGMENTS
We would like to express our thanks to Hanno, Jonathan and Quirin for reporting the problem to us, and to Andrew Ayer for providing https://caatestsuite.com/.

Comment 5

[Michael Kliewe]

@Rob Stradling
Thanks for the incident report.

You could have avoided the problem at all, because I reported the problem to you already on 07 Aug 2017 17:32:20 +0000 and on 08 Aug 2017 10:35:07 +0000 via your Online Chat. First your Chat support did not understand my report, then the support did not answer anymore. I asked for a follow-up to my report, but nothing happened.

On the second day, I tried it again. Again the chat support misunderstood my report and the potentially high impact, and did not forward it to the security team or developer team. I was too lazy to try it a third time, because back then it was "just" a wrong announcement in your knowledgebase, but I knew that it could be problematic in September. 
https://support.comodo.com/index.php?/Knowledgebase/Article/View/1204/1/caa-record---certification-authority-authorization
"All CA's will be mandated to check CAA DNS records starting in late 2017. Comodo, however, has been supporting this on ALL certificates for the last 12+ months. "

I was referring to that sentence, because CAA checking was not mandatory back in August.

Sadly your chat support missed those two chances to detect and fix the problem before CAA checking became mandatory.

Michael

Comment 6

[Quirin Scheitle]

Hi Rob,

Thank you for following up that quickly and effectively. 

Please also feel free to retry for my domains -- I myself can not retest as "trial certificates have already been obtained" ;)

The fact that your tried and tested system silently degraded also makes me think whether we as a community should maintain a list of test domains that CAs can continuously test against. This would also very much help to translate the CAB/RFC language into specific yes/no cases. 

My test cases are documented under https://github.com/quirins/caa-test.

Kind regards
Quirin

Comment 7

[James Burton]

Hi,

We should included in every certificate issued a CA reporting email address in the certificate subject OU. This CA reporting email address would help people who find security vulnerabilities to report the issues to the right people in charge. If this was  implemented already I think This bug would have never existed because the issue would have been fixed.

James Burton

Comment 8

[Rob Stradling]

Michael,
Thank you for attempting to report this problem to us earlier.  I'm sorry our 1st Line Support team did not comprehend and escalate your report, but I hope you understand that different Comodo employees have different skillsets.

The CABForum BRs contain the concept of a Certificate Problem Report.  Section 4.9.3 says:
"The CA SHALL provide Subscribers, Relying Parties, Application Software Suppliers, and other third parties with clear instructions for reporting suspected Private Key Compromise, Certificate misuse, or other types of fraud, compromise, misuse, inappropriate conduct, or any other matter related to Certificates. The CA SHALL publicly disclose the instructions through a readily accessible online means."

Section 4.9.2 of our CPS says:
"Other parties may report suspected Private Key Compromise, Certificate misuse, or other types of fraud, compromise, misuse, inappropriate conduct, or any other matter related to Certificates, in the first instance, by email to [e-mail]."

Our appropriately skilled staff will respond to Certificate Problem Reports when they become aware of them.  Sending an email to [e-mail] is the way to ensure that that happens.

Comment 9

[Rob Stradling]

Quirin,
Thanks, I'll definitely take a look at your test cases.

Comment 10

[Rob Stradling]

James,
We already include the URL of our CPS in certificates, and our CPS already indicates how to submit a Certificate Problem Report.

In the April 2017 CA Communication, Mozilla gathered together a central list of CAs' Problem Reporting Mechanisms:
https://ccadb-public.secure.force.com/mozillacommunications/CACommResponsesOnlyReport?CommunicationId=a05o000003WrzBC&QuestionId=Q00028

I don't think further abuse of the Subject:OU field is where we want to be heading, TBH.  However, it could prove useful for the community to have a discussion about how to increase awareness of the correct methods for submitting Certificate Problem Reports to CAs.

Comment 11

[James Burton]

(In reply to Rob Stradling from comment #10)

> James,
> We already include the URL of our CPS in certificates, and our CPS already
> indicates how to submit a Certificate Problem Report.
> In the April 2017 CA Communication, Mozilla gathered together a central list
> of CAs' Problem Reporting Mechanisms:
> https://ccadb-public.secure.force.com/mozillacommunications/
> CACommResponsesOnlyReport?CommunicationId=a05o000003WrzBC&QuestionId=Q00028

Rob,

We can't expect the average normal technical person to start rummaging through CPSs or etc to find reporting mechanism used by CAs. It's an unnecessary hurdle to jump over and damages security. We should be developing a really easy way for reporters to report security related issues to CAs. In Comodo's case, the page here (https://www.comodo.com/about/comodo-agreements.php) should've had an email address located in a section named "Reporting" above notices.

> I don't think further abuse of the Subject:OU field is where we want to be
> heading, TBH.  However, it could prove useful for the community to have a
> discussion about how to increase awareness of the correct methods for
> submitting Certificate Problem Reports to CAs.

All CAs should be required to include some type of security related reporting email address in their certificates. One example was including it in the subject OU but like you said before its a misuse of the field. Another way we could include a reporting email address is in the Certificate Policies/User Notice section.

James

Comment 12

[Rob Stradling]

James,

"We can't expect the average normal technical person to start rummaging through CPSs"
  - BRs section 2.2 says that CPSs "MUST be structured in accordance with either RFC 2527 or RFC 3647", so in theory there should be a very small number of sections to look at in each CPS to find the details of the reporting mechanism.  Now I have to admit that I dislike reading CPSs as much as the next guy, but I don't really buy the implied argument that a (typically) PDF document is significantly harder to read than an HTML page.

"We should be developing a really easy way for reporters to report security related issues to CAs"
  - Well, I'll do my bit: https://github.com/crtsh/certwatch_db/issues/36

"All CAs should be required to include some type of security related reporting email address in their certificates"
  - I think that's a discussion to be had on m.d.s.p. or at CABForum rather than in this bug.

Comment 13

[Gervase Markham [:gerv]]

(In reply to Rob Stradling from comment #12)
> "All CAs should be required to include some type of security related
> reporting email address in their certificates"
>   - I think that's a discussion to be had on m.d.s.p. or at CABForum rather
> than in this bug.

If it helps cut it short: I'm not in favour of such a requirement. The cert is not the right place for this information.

Rob: thank you for your detailed report. Perhaps you could have a moan at the BIND developers for removing a command-line option in a micro-point release...

Gerv

Comment 14

[Rob Stradling]

Hi Gerv.  Upon closer investigation just now, I (re)discovered that dig's +sigchase option is only available if BIND has been built with -DDIG_SIGCHASE.

We specified -DDIG_SIGCHASE when we deployed BIND 9.10.1-P2 for our original CAA checking implementation, but presumably our ops team did not specify -DDIG_SIGCHASE when they chose to deploy BIND 9.10.5-P1 at the time of the server upgrade.

So the +sigchase option wasn't removed by the BIND developers in a micro-point release.

It would be nice if the BIND developers would set -DDIG_SIGCHASE by default though, to avoid the nasty trap that our ops team fell into.  https://bugs.archlinux.org/task/47033 describes another victim.

Comment 15

[Simon]

I just tested this on the domain ciccio.panza.tern.al that has a CAA record on tern.al:
0 issue "letsencrypt.org"
0 iodef "mailto:[e-mail]"
Comodo issued a valid cert for https://ciccio.panza.tern.al/

I got no mail from comodo on [e-mail]

Comment 16

[Rob Stradling]

Hi Simon.  Thanks for testing.  For how long has tern.al had that CAA record?

We performed CAA queries for ciccio.panza.tern.al, panza.tern.al, tern.al and al about half a minute before issuing that DV certificate (https://crt.sh/?id=218137342).  Those queries found no CAA records, and since we'd already verified control of the domain, we issued the certificate.

Below is the DNS reply we received when we performed a CAA query for tern.al:


;; opcode: QUERY, status: NOERROR, id: 49706
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;tern.al.	IN	 CAA

;; AUTHORITY SECTION:
tern.al.	86400	IN	SOA	ns1.he.net. hostmaster.he.net. 2017092505 10800 1800 604800 86400

;; ADDITIONAL SECTION:

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 4096

Comment 17

[Simon]

Hello Rob,
I added the CAA record a few days ago, and tested if the record was there from my local machine with dig, and with https://caatest.co.uk/tern.al before submitting the request for the cert.
But I can not exclude that the change did not propagate to ns1.he.net, but propagated correctly to ns[2-5].he.net, those servers are not under my control.

Comment 18

[Terry Burton]

CT logs indicate the misissuance by cPanel/Comodo of the following certificate in violation of the parent label's CAA policy:

https://crt.sh/?id=266399724

The zone is DNSSEC signed, consistent across all nameservers and the CAA records have remained in this state since early November:

le.ac.uk.	CAA	0 issue "quovadisglobal.com\; policy=ev"
le.ac.uk.	CAA	0 issuewild "\;"
le.ac.uk.	CAA	0 iodef "mailto:[e-mail]"

Following the indicated process of emailing [e-mail] has merely resulted in the creation of an entry (#EWT-314-62006) in the ticketing system (that I have reclassified as urgent) which has now been unacknowledged for more than 24 hours.

I raise this here as it seems that there are ongoing technical and triage process failures.

> Our appropriately skilled staff will respond to Certificate Problem Reports when they become aware of them.  Sending an email to [e-mail] is the way to ensure that that happens.

Comment 19

[Rob Stradling]

Hi Terry.  I saw your email to [e-mail] and it was on my radar to reply to it yesterday, but unfortunately I ran out of time to do that.

https://crt.sh/?id=266399724 was misissued due to the same bug mentioned at https://bugzilla.mozilla.org/show_bug.cgi?id=1420873#c17.  We patched this bug yesterday.  I'm just about to scan our logs for all misissuances due to this bug, then write an incident report, and then engage our Support staff to notify the affected customers and to get the affected certs revoked.

Sorry this hasn't all happened within 24 hours.  24 hours is hard.

Comment 20

[Terry Burton]

Thanks for the update.

In the meantime the certificate has been revoked without reference to any follow on root cause analysis. However, it's good to know this has happened and I look forward (academically) to reading the report.

> Sorry this hasn't all happened within 24 hours.  24 hours is hard.

Sure. A basic acknowledgement goes a long way.

Sincere thanks for your effort.

Comment 21

[Gervase Markham [:gerv]]

Terry: can we please have one bug per issue, rather than adding new issues to an already-closed bug? Thanks :-) Could either you or Rob, if he gets there first, file one so Rob has somewhere to put his incident report?

Gerv

Comment 22

[Terry Burton]

> Terry: can we please have one bug per issue, rather than adding new issues to an already-closed bug? Thanks :-) Could either you or Rob, if he gets there first, file one so Rob has somewhere to put his incident report?

Sure: https://bugzilla.mozilla.org/show_bug.cgi?id=1423624

Thanks.