Closed Bug 1423624 Opened 8 years ago Closed 8 years ago

Comodo: CAA misissuances due to race condition

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: tez, Assigned: rob)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36 Steps to reproduce: This is an individual bug report entry for the mis-issuance confirmed here: https://bugzilla.mozilla.org/show_bug.cgi?id=1398545#c18 Pending incident report from Comodo.
Thanks Terry. INCIDENT REPORT On Tuesday 28th November, James Billingham posted a comment [1] to bug #1420873 to report a CAA misissuance for https://crt.sh/?id=266205228. James didn't use our established Problem Reporting Mechanism (see [2]), but I became aware of his comment when I responded to bug #1420873 on Friday 1st December. TRIAGING A review of our logs for this certificate showed that our CAA checker had (correctly) determined that we were Not Authorized to issue https://crt.sh/?id=266205228 due to the CAA RRset described at [2]. Somewhat baffled, I looked at our certificate issuance code, but I could not see any way in which a "Not Authorized" result could lead to issuance. WHAT WENT WRONG It was only when I looked closely at the timestamps of each event relating to this certificate's CAA checks and issuance that it became clear what had happened: 2017-11-28 16:59:13: CAA lookup for web-seo.prod.ext.cuvva.co => Empty 2017-11-28 16:59:13: CAA lookup for prod.ext.cuvva.co => Empty 2017-11-28 16:59:15: CAA lookup for ext.cuvva.co => Empty 2017-11-28 16:59:15: CAA lookup for cuvva.co => Not Authorized 2017-11-28 16:59:13: TBSCertificate was generated. Our certificate issuance code looks at the result of only the most recent CAA lookup pertaining to the certificate request. From the timestamps, it's clear that only the 1st (and perhaps also the 2nd) CAA lookup had occurred at the time when the certificate issuance code processed this certificate request. Therefore, it must've seen an "Empty" result and proceeded to issue. HOW DID THIS HAPPEN? In our initial CAA checking implementation (discussed at [3]), all of the CAA checks for a certificate occurred in a single SQL transaction. (The checks were driven from a database stored procedure, which is transaction-based by default). Therefore, the certificate issuance code never saw a partially complete CAA tree climb, and so this race condition never occurred. In our (unfortunately but necessarily) hurried switch to a Go-based CAA checking implementation (again, discussed at [3]), we overlooked the fact that our certificate issuance code was assuming that CAA tree climbing was an atomic operation. (Unlike the database stored procedures in our previous CAA checking implementation, Go's database interfaces aren't transaction-based by default). ACTION TAKEN TO ADDRESS THE PROBLEM Having diagnosed the bug, we immediately prepared and tested a bugfix that causes our certificate issuance code to treat an "Empty" result for a non-TLD as "tree climbing still in progress, so put this certificate request on hold for now" rather than as permission to issue. The bugfix was deployed on Monday 5th December at 16:53 UTC. NEXT STEPS Since we have logs of the CAA checks that we performed but which our certificate issuance code did not wait for, we will scan those logs to determine which other certificates were misissued as a result of this bug. I will post the results of this scan to this bug and then we will take appropriate action. OTHER INFORMATION Terry Burton used our established Problem Reporting Mechanism to report https://crt.sh/?id=266399724 (the certificate mentioned in comment #0) as a CAA misissuance. We received Terry's report on 4th December at 20:49 UTC. I investigated and found that this certificate was misissued due to the same race condition described by this incident report. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1420873#c2 [2] https://ccadb-public.secure.force.com/mozillacommunications/CACommResponsesOnlyReport?CommunicationId=a05o000003WrzBC&QuestionId=Q00028 [3] https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg08054.html
Assignee: kwilson → rob
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Summary: cPanel/Comodo: CAA Misissuance → Comodo: CAA misissuances due to race condition
Whiteboard: [ca-compliance]
We've identified 270 certificates that were misissued due to the race condition bug. We will notify the affected customers and get these certificates revoked. These certs are also being tracked at https://misissued.com/batch/36/ https://crt.sh/?id=238660924 https://crt.sh/?id=242989792 https://crt.sh/?id=243326645 https://crt.sh/?id=244794857 https://crt.sh/?id=246292407 https://crt.sh/?id=248126260 https://crt.sh/?id=248189889 https://crt.sh/?id=249282590 https://crt.sh/?id=250201717 https://crt.sh/?id=251677426 https://crt.sh/?id=252838784 https://crt.sh/?id=252891353 https://crt.sh/?id=252891445 https://crt.sh/?id=258693121 https://crt.sh/?id=262173538 https://crt.sh/?id=262289567 https://crt.sh/?id=262289580 https://crt.sh/?id=263517638 https://crt.sh/?id=264779603 https://crt.sh/?id=265015636 https://crt.sh/?id=265015804 https://crt.sh/?id=265411239 https://crt.sh/?id=265582033 https://crt.sh/?id=265735328 https://crt.sh/?id=266205228 https://crt.sh/?id=266233432 https://crt.sh/?id=266399724 https://crt.sh/?id=266848864 https://crt.sh/?id=266984442 https://crt.sh/?id=267714773 https://crt.sh/?id=267915618 https://crt.sh/?id=268463602 https://crt.sh/?id=268533711 https://crt.sh/?id=269285891 https://crt.sh/?id=269977804 https://crt.sh/?id=269979574 https://crt.sh/?id=270496010 https://crt.sh/?id=271612518 https://crt.sh/?id=272619726 https://crt.sh/?id=272619728 https://crt.sh/?id=272619729 https://crt.sh/?id=272619735 https://crt.sh/?id=272619742 https://crt.sh/?id=272619743 https://crt.sh/?id=272619744 https://crt.sh/?id=272619745 https://crt.sh/?id=272619746 https://crt.sh/?id=272619747 https://crt.sh/?id=272619748 https://crt.sh/?id=272619749 https://crt.sh/?id=272619750 https://crt.sh/?id=272619751 https://crt.sh/?id=272619754 https://crt.sh/?id=272619757 https://crt.sh/?id=272619760 https://crt.sh/?id=272619762 https://crt.sh/?id=272619764 https://crt.sh/?id=272619765 https://crt.sh/?id=272619768 https://crt.sh/?id=272619770 https://crt.sh/?id=272619772 https://crt.sh/?id=272619774 https://crt.sh/?id=272619776 https://crt.sh/?id=272619778 https://crt.sh/?id=272619781 https://crt.sh/?id=272619783 https://crt.sh/?id=272619786 https://crt.sh/?id=272619789 https://crt.sh/?id=272619792 https://crt.sh/?id=272619795 https://crt.sh/?id=272619796 https://crt.sh/?id=272619800 https://crt.sh/?id=272619805 https://crt.sh/?id=272619806 https://crt.sh/?id=272619809 https://crt.sh/?id=272619811 https://crt.sh/?id=272619814 https://crt.sh/?id=272619816 https://crt.sh/?id=272619819 https://crt.sh/?id=272619820 https://crt.sh/?id=272619822 https://crt.sh/?id=272619824 https://crt.sh/?id=272619828 https://crt.sh/?id=272619831 https://crt.sh/?id=272619834 https://crt.sh/?id=272619839 https://crt.sh/?id=272619843 https://crt.sh/?id=272619845 https://crt.sh/?id=272619848 https://crt.sh/?id=272619851 https://crt.sh/?id=272619853 https://crt.sh/?id=272619855 https://crt.sh/?id=272619859 https://crt.sh/?id=272619861 https://crt.sh/?id=272619864 https://crt.sh/?id=272619867 https://crt.sh/?id=272619870 https://crt.sh/?id=272619873 https://crt.sh/?id=272619876 https://crt.sh/?id=272619878 https://crt.sh/?id=272619880 https://crt.sh/?id=272619884 https://crt.sh/?id=272619886 https://crt.sh/?id=272619887 https://crt.sh/?id=272619891 https://crt.sh/?id=272619893 https://crt.sh/?id=272619897 https://crt.sh/?id=272619898 https://crt.sh/?id=272619901 https://crt.sh/?id=272619904 https://crt.sh/?id=272619907 https://crt.sh/?id=272619909 https://crt.sh/?id=272619911 https://crt.sh/?id=272619914 https://crt.sh/?id=272619918 https://crt.sh/?id=272619920 https://crt.sh/?id=272619925 https://crt.sh/?id=272619926 https://crt.sh/?id=272619928 https://crt.sh/?id=272619931 https://crt.sh/?id=272619933 https://crt.sh/?id=272619937 https://crt.sh/?id=272619939 https://crt.sh/?id=272619941 https://crt.sh/?id=272619944 https://crt.sh/?id=272619946 https://crt.sh/?id=272619947 https://crt.sh/?id=272619950 https://crt.sh/?id=272619952 https://crt.sh/?id=272619955 https://crt.sh/?id=272619958 https://crt.sh/?id=272619961 https://crt.sh/?id=272619963 https://crt.sh/?id=272619966 https://crt.sh/?id=272619969 https://crt.sh/?id=272619970 https://crt.sh/?id=272619974 https://crt.sh/?id=272619976 https://crt.sh/?id=272619980 https://crt.sh/?id=272619982 https://crt.sh/?id=272619984 https://crt.sh/?id=272619986 https://crt.sh/?id=272619988 https://crt.sh/?id=272619992 https://crt.sh/?id=272619995 https://crt.sh/?id=272619998 https://crt.sh/?id=272620000 https://crt.sh/?id=272620002 https://crt.sh/?id=272620005 https://crt.sh/?id=272620008 https://crt.sh/?id=272620011 https://crt.sh/?id=272620014 https://crt.sh/?id=272620017 https://crt.sh/?id=272620019 https://crt.sh/?id=272620022 https://crt.sh/?id=272620025 https://crt.sh/?id=272620026 https://crt.sh/?id=272620030 https://crt.sh/?id=272620032 https://crt.sh/?id=272620036 https://crt.sh/?id=272620039 https://crt.sh/?id=272620041 https://crt.sh/?id=272620043 https://crt.sh/?id=272620045 https://crt.sh/?id=272620049 https://crt.sh/?id=272620052 https://crt.sh/?id=272620056 https://crt.sh/?id=272620058 https://crt.sh/?id=272620061 https://crt.sh/?id=272620062 https://crt.sh/?id=272620066 https://crt.sh/?id=272620068 https://crt.sh/?id=272620073 https://crt.sh/?id=272620075 https://crt.sh/?id=272620078 https://crt.sh/?id=272620082 https://crt.sh/?id=272620087 https://crt.sh/?id=272620091 https://crt.sh/?id=272620094 https://crt.sh/?id=272620097 https://crt.sh/?id=272620101 https://crt.sh/?id=272620104 https://crt.sh/?id=272620108 https://crt.sh/?id=272620110 https://crt.sh/?id=272620114 https://crt.sh/?id=272620115 https://crt.sh/?id=272620121 https://crt.sh/?id=272620123 https://crt.sh/?id=272620126 https://crt.sh/?id=272620130 https://crt.sh/?id=272620131 https://crt.sh/?id=272620133 https://crt.sh/?id=272620136 https://crt.sh/?id=272620138 https://crt.sh/?id=272620140 https://crt.sh/?id=272620143 https://crt.sh/?id=272620148 https://crt.sh/?id=272620152 https://crt.sh/?id=272620155 https://crt.sh/?id=272620158 https://crt.sh/?id=272620168 https://crt.sh/?id=272620169 https://crt.sh/?id=272620173 https://crt.sh/?id=272620175 https://crt.sh/?id=272620177 https://crt.sh/?id=272620181 https://crt.sh/?id=272620186 https://crt.sh/?id=272620188 https://crt.sh/?id=272620193 https://crt.sh/?id=272620198 https://crt.sh/?id=272620201 https://crt.sh/?id=272620205 https://crt.sh/?id=272620208 https://crt.sh/?id=272620212 https://crt.sh/?id=272620215 https://crt.sh/?id=272620219 https://crt.sh/?id=272620224 https://crt.sh/?id=272620227 https://crt.sh/?id=272620231 https://crt.sh/?id=272620233 https://crt.sh/?id=272620238 https://crt.sh/?id=272620242 https://crt.sh/?id=272620248 https://crt.sh/?id=272620251 https://crt.sh/?id=272620253 https://crt.sh/?id=272620258 https://crt.sh/?id=272620262 https://crt.sh/?id=272620268 https://crt.sh/?id=272620271 https://crt.sh/?id=272620275 https://crt.sh/?id=272620281 https://crt.sh/?id=272620286 https://crt.sh/?id=272620289 https://crt.sh/?id=272620297 https://crt.sh/?id=272620298 https://crt.sh/?id=272620302 https://crt.sh/?id=272620303 https://crt.sh/?id=272620306 https://crt.sh/?id=272620307 https://crt.sh/?id=272620310 https://crt.sh/?id=272620313 https://crt.sh/?id=272620316 https://crt.sh/?id=272620319 https://crt.sh/?id=272620322 https://crt.sh/?id=272620324 https://crt.sh/?id=272620327 https://crt.sh/?id=272620330 https://crt.sh/?id=272620334 https://crt.sh/?id=272620337 https://crt.sh/?id=272620339 https://crt.sh/?id=272620340 https://crt.sh/?id=272620342 https://crt.sh/?id=272620346 https://crt.sh/?id=272620351 https://crt.sh/?id=272620354 https://crt.sh/?id=272620357 https://crt.sh/?id=272620360 https://crt.sh/?id=272620364 https://crt.sh/?id=272620367 https://crt.sh/?id=272620369 https://crt.sh/?id=272620373 https://crt.sh/?id=272620376 https://crt.sh/?id=272620377 https://crt.sh/?id=272620380 https://crt.sh/?id=272620384 https://crt.sh/?id=272620387 https://crt.sh/?id=272620390 https://crt.sh/?id=272620393 https://crt.sh/?id=272620396 https://crt.sh/?id=272620399
All 270 certificates listed at https://misissued.com/batch/36/ have now been revoked.
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [dv-misissuance]
You need to log in before you can comment on or make changes to this bug.