1492006 - Sectigo: Failure to revoke within 24 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

The following message was sent to the mozilla.dev.security.polyc list:
==========================
Hello, I am the domain owner of debigare.com. I would like to make you aware that Comodo CA took more than 5 days to revoke certificates they had signed for my domain and subdomains after requesting them to do through their sslabuse email address, past the 24 hours maximum mentioned in the Baseline Requirements as stipulated in section 4.9.1.1.

For context, I was previously using Cloudflare's Universal SSL feature, but disabling it did not revoke the old certificates that had not yet expired, but simply removed them from its system, and some of the certificates were still valid for more than 6 months.

I first attempted to contact Cloudflare's support to ask them to revoke the certificates themselves on September 6 at 7:43 UTC. This only led to irrelevant responses and confused customer support agents that had no idea what I was talking about, and this appeared to go nowhere. I eventually got a response from them on September 11 at 5:53 UTC that they would request CAs to perform the revocation, but that was after I did so myself, and I never got a status report back afterwards.

There were two CAs affected by this issue. The vast majority of certificates were signed by Comodo CA, and the rest by DigiCert. I did not run into any issues with DigiCert (they in fact proactively checked with Cloudflare my claim and revoked the certificates before I even had the chance to attempt their domain ownership challenge), but Comodo CA was another story entirely.

My first request to Comodo CA to revoke the certificates for debigare.com and all of its subdomains was on September 8 at 4:37 UTC. I did not get a reply until September 9 at 15:53 UTC stating that the certificates have been revoked. Not only was this past the 24 hours requirement, but it was also false, as only the most recent certificates had been revoked, not all of them. I mentioned to them their mistake on September 10 at 5:55 UTC with a full list of affected certificates just in case my initial request was unclear to them, and never got a reply back. I did, however, notice that the certificates eventually got revoked on September 13 at 16:04 UTC according to their Certificate Transparency logs, a fact that I only discovered on September 15. Assuming the log is correct, that would be a delay of more than 3 days after notifying them of the incomplete revocation, more than 5 days after my initial request to them, and more than a week until I finally noticed the problem was fixed. It's also worth noting that throughout this entire series of events, Comodo CA never requested proof of domain ownership beyond the evidence I initially provided, so that cannot explain the delays.

One detail that I'm not sure about is why my initial evidence for domain ownership was apparently sufficient for Comodo CA but not for DigiCert. On this regard, the only evidence I provided to both of them was that the email address I used to contact them matched the contact information on my website. (My emails were protected with SPF, DKIM and DMARC for authenticity.) For some reason, DigiCert believed that this evidence did not met the Baseline Requirements for my request, a claim that I am currently unable to verify as I cannot find anything of the sort in them.

You can read the full story on my blog, which I hope will be sufficient to prove my identity: https://www.debigare.com/4-unexpected-issues-i-encountered-upon-switching-web-hosts/

I can also provide a full copy of the email exchange I had with Comodo CA as evidence on request.

Guillaume Fortin-Debigaré
=========================
Since Comodo did eventually reply and revoke these certificates, just not within 24 hours, I believe we have evidence of a BR violation.

Please provide an incident report, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the corresponding mozilla.dev.security.policy thread and added to this bug.

Comment 1

[Rob Stradling]

Robin, please respond to this.  Thanks.

Comment 2

[Wayne Thayer]

Robin: ping?

Comment 3

[Robin Alden]

Using the incident report template from
https://wiki.mozilla.org/CA/Responding_To_An_Incident :

* How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Sat 17-SEP-18 02:31 UTC
The revocation requester posted an email to [e-mail] 
https://groups.google.com/d/msg/mozilla.dev.security.policy/rAgAbZBIG0o/CzrGuZJsBgAJ
raising the complaint that a revocation request had taken too long to process.

* A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Sat 08-SEP-18 07:38 UTC	 email from the requester to [e-mail] requesting that we:
"revoke all valid and unexpired Comodo TLS certificates for debigare.com, www.debigare.com and *.debigare.com that were generated by Cloudflare"
Sun 09-SEP-18 15:52:04 UTC  https://crt.sh/?id=675400028 was revoked.
Sun 09-SEP-18 16:53 UTC  email from [e-mail] informing requester of revocation.
Mon 10-SEP-18 06:55 UTC  email from requester to [e-mail]
Thank you! Unfortunately, it appears that only a subset of them have been revoked, at least through OCSP.
To clarify, I'm expecting all entries from both linked lists below with "O=COMODO CA Limited" to be revoked.
https://crt.sh/?Identity=debigare.com&exclude=expired
https://crt.sh/?Identity=%.debigare.com&exclude=expired
Thu 13-Sep-18 16:04:56 UTC  all remaining certificates revoked.
https://crt.sh/?serial=008086E28EF66FFEF6975B9F52FA3EE991
https://crt.sh/?serial=0084926FDB69FC3CF33677F30AFD03AF9A
https://crt.sh/?serial=00885019F755DC2A13A578DBB3C8FE3B6A
https://crt.sh/?serial=008C8194AA5C8B5BD510A09284C3468536
https://crt.sh/?serial=008F287A6B83C7F0DE39087CBC64F8742F
https://crt.sh/?serial=009280DFBF14DCE4F6BB62FD6A134B968C
https://crt.sh/?serial=0095598F63CB2DFEE9F309E85BA8B8BB6D
https://crt.sh/?serial=00981EC11093E4A5F87CE5D05188CBB069
https://crt.sh/?serial=009D0FFD5A2E9137C94A65D2BBE0B1241F
https://crt.sh/?serial=009DA7A06B68482BD961EF9837D49B3221
https://crt.sh/?serial=009E41DB9BD7D714C7BDD0A0C70681C77F
https://crt.sh/?serial=009E8C23BE61E2EE18D7714874FBF8FA6A
https://crt.sh/?serial=00A00ADF9AB6BD11160126F3BFBC9BA9DB
https://crt.sh/?serial=00A975BF18C22BA0799CCE6141BE7DF1D7
https://crt.sh/?serial=00B15B1BEF6000F9457797A6FA18B38CB4
https://crt.sh/?serial=00B2F35F94A013C5C5C3FF52C1249B72F2
https://crt.sh/?serial=00B68F6C8BE167B0999905B52F78AEF874
https://crt.sh/?serial=00BD14520F1D29C71308BD5FFE616FF6DB
https://crt.sh/?serial=00C12A51F5ADD9201A31A63E53323E376B
https://crt.sh/?serial=00C1D6FD8DFF4CA95D6A96CDB9F993E67D
https://crt.sh/?serial=00C503B7F702CBBB0228EAB67D5F726769
https://crt.sh/?serial=00C9EA9AD0ACE22529D29670D99C3F9482
https://crt.sh/?serial=00CE9F73A3B3D770878C118AC23D41FD66
https://crt.sh/?serial=00D0A911AAD6D537CF9280B1D65FD0F41D
https://crt.sh/?serial=00D15597983C0594C163995129FEE013ED
https://crt.sh/?serial=00D3C57DA7856EF52F5E8712510523FE7A
https://crt.sh/?serial=00D432135D5A51859E9013B9B7B556C361
https://crt.sh/?serial=00D592156D6F614720D67316E0044FEA22
https://crt.sh/?serial=00D92B90893CEAF704062BFDCA1E35A524
https://crt.sh/?serial=00DAFB773443D7576F22D86D2E0BF1FB5D
https://crt.sh/?serial=00DD78D7A3DC4DC14D027B4CE3CB03D2DC
https://crt.sh/?serial=00DFFDDC31AB3917681746E45E575F326E
https://crt.sh/?serial=00E256BFBD0DFEA98C73C8A02A29651E05
https://crt.sh/?serial=00E91BB0DD3A8B2587D49EC2138852A496
https://crt.sh/?serial=00E9BD910E62720AC13FB38A1625300963
https://crt.sh/?serial=00F948628429F6EBF4E3FE2F82224DECDA
https://crt.sh/?serial=00FB508DCADA6FF5857E45E3585D6C2EFE
https://crt.sh/?serial=03B82A3DD40D41FDE38B5C8132B5DA83
https://crt.sh/?serial=04BE68646CBD1524A59B2B282F784CAD
https://crt.sh/?serial=09578B3288F913841CD4E3323040DC20
https://crt.sh/?serial=0BA5C7E768F4BFE109F7097F4D99A676
https://crt.sh/?serial=0D6A440D4174580E62C277CEA5A35FF1
https://crt.sh/?serial=0DC7922BA7009E2CDEAEEE01C23E9715
https://crt.sh/?serial=13298344BB97CAC40DBA6A0AAC1BC05C
https://crt.sh/?serial=182A915CC1D49AB8E28900B9ABB81D1E
https://crt.sh/?serial=18AAEB68C80A4A9BFBED9E48D5B7072E
https://crt.sh/?serial=1CC6274880580C53BC56BEDB0863D4CB
https://crt.sh/?serial=1D24CEA27FDB4429A763EEC7B9373927
https://crt.sh/?serial=22628158DB2ABB8E662F67D72ADA6AEF
https://crt.sh/?serial=235D539C3E257C901E2F82B4CFC5AA8D
https://crt.sh/?serial=242A0EE87BBD0F58B202D28610A7E063
https://crt.sh/?serial=29376A49CC6B328C0AA58A39459CFA98
https://crt.sh/?serial=34F852D6D2AD3A9981FD20249FF9565E
https://crt.sh/?serial=361692A7DC7FA185B309DF7655E8946A
https://crt.sh/?serial=365B5F5D6208B4A8CD1CDFF80396A0FB
https://crt.sh/?serial=36D71924E2BB392FDC9F5035C41755CF
https://crt.sh/?serial=37BBDDAF38C8D3B3EE7197050C4C269C
https://crt.sh/?serial=37DE789AB9494389282B2AB8112C6C46
https://crt.sh/?serial=39A90384F06C09F95BDF8D7A09A8D3A7
https://crt.sh/?serial=39D0EC51A7542EE58775132647C7369B
https://crt.sh/?serial=3DA154E80A0AFAFBB55BC8FC1CD84039
https://crt.sh/?serial=4A4B3AD6DCE0BEAAE93746259AB32160
https://crt.sh/?serial=4DE5D17720EA0FA471F31CB3033C4FFB
https://crt.sh/?serial=530634B5962159F406B772D27487303B
https://crt.sh/?serial=577D81E89C7AF0976ABC354F3BCADF89
https://crt.sh/?serial=5D795DF91BAD36B7C13003BA26CFF528
https://crt.sh/?serial=5DC9A3BBE0486B4085403F883740E194
https://crt.sh/?serial=62CF71D11DDD35C2FE35F85E4A3ABE7D
https://crt.sh/?serial=64F3A9C84FEAAB61676E78FFB457A1EA
https://crt.sh/?serial=663D95BE405ABB6C0D9755E16EDF8B12
https://crt.sh/?serial=692789F36D1E22CFE85D7DBC472E008E
https://crt.sh/?serial=713B6D14355A678393D477B518408776
https://crt.sh/?serial=717D4A11836EB074716AEB9F5BE7D06B
https://crt.sh/?serial=750C6F100C6124E09E2767BB7C4BFFA0
https://crt.sh/?serial=7B4E92FD2CEC596BA34293BD0ABEE2EC

* Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
NA

* A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
One certificate was revoked 32 hours and 14 minutes after the request was received.
The remainder of the certificates were revoked 83 hours and 20 minutes after the clarification was received.

* The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
The list of revoked certificates is above.

* Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

For the first certificate, our rep would have checked with Cloudflare that the requester was permitted to request the revocation.  
The BRs say in section 4.9.1.1:
> The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs:
> 1. The Subscriber requests in writing that the CA revoke the Certificate;
The requester was not the subscriber.
Because of the time taken to check with an external party, plus the time taken by us to process the request, more than 24 hours elapsed from the requester's email.
We followed our usual process flow in revoking a single certificate in response to the request when we should have confirmed with the requester that multiple certificates were involved.

For the remaining certificates, we had to identify in our systems the list of certificates to be revoked and then revoke them.  The request was potentially ambiguous and processing it fell outside our usual work flow and it was passed to other staff for processing.  The urgency of the revocation request was not adequately communicated as the request was passed along.

* List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

1) We have re-iterated to our staff the urgency of revocation requests being processed to completion.
2) We are re-evaluating our CPS and website language around revocation requests.  
We help our customers to help us revoke their certificates most easily and quickly when the revocation requests are clear, well defined and completely unambiguous.

Regards
Robin Alden
Sectigo Limited

Comment 4

[Wayne Thayer]

Robin: thank you for the incident report.

The reporter has requested clarification on what made his request ambiguous and difficult to process [1]?

Also, please update this bug with the results of action #2 in the incident report.


[1] https://groups.google.com/d/msg/mozilla.dev.security.policy/rAgAbZBIG0o/CzrGuZJsBgAJ

Comment 5

[Robin Alden]

(In reply to Wayne Thayer [:wayne] from comment #4)
> The reporter has requested clarification on what made his request ambiguous
> and difficult to process [1]?
> [1]
> https://groups.google.com/d/msg/mozilla.dev.security.policy/rAgA
> CzrGuZJsBgAJ

I did not mean to criticize the requester's report.  His request was clear.
The ambiguity to which I referred was meant in the sense that it applied to our customer support representatives who responded to the request.  Their typical process flow in revoking is to identify a certificate and revoke it.
In this case we had to determine the entire list of affected certificates, check that the requester was the domain owner, and then revoke the certificates.  

> Also, please update this bug with the results of action #2 in the incident
> report.

We have not yet concluded the re-evaluation about our revocation process.

So far it seems clear that we would have saved ourselves a bunch of pain AND have more quickly enabled the requester to achieve his aim of revoking these certificates had we provided an automated mechanism by which he could have triggered the revocation himself.
We are defining a set of mechanisms which will:
a) allow domain owners to command revocation of certificates that include their domains
b) allow anyone to command revocation of certificates for which they control the private key.
These would operate in addition to the existing mechanisms that allow our customers to revoke certificates they have ordered.

Comment 6

[Ryan Sleevi]

Robin: Do you have updates from what's described in Comment #5? Do you have a date in which you'll be able to provide updates? Otherwise, it may be useful to switch to weekly updates, as described in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

Comment 7

[Ryan Sleevi]

Robin: Can you provide an update?

Comment 8

[Robin Alden]

We have the automated revocation mechanisms in outline form. I will post substantial details here by Thursday Jan 24th.

Comment 9

[Wayne Thayer]

Robin: it has been almost a month since you stated in comment #8 that you would provide more information.

Comment 10

[Robin Alden]

Our Self-service revocation portal will support these methods:

#1 If the certificate you want revoked was ordered by you directly from Sectigo (or previously from Comodo CA), log into your account with us and select the certificate to be revoked.
This mechanism is in place and has been for a long time.

#2 If the certificate you want revoked is one for which you hold the private key either because you generated the key in the first place or because you found the key on the internet, you may do one of the following:

#2a create a revocation token by appending the certificate serial number to the string "REVOKE:".
E.g. "REVOKE:23:b5:72:50:18:7d:2f:83:03:9e:27:09:34:0b:4d:eb"
Use the private key to generate a SHA-256 based signature over the revocation token. Send us the revocation token and the signature and we will revoke the certificate.
Instructions on the use of openssl to accomplish this will be given in the portal.

#2b send us the Certificate and the private key and we will revoke the certificate.

#3 If the certificate you want revoked has a domain that you control in its subject (in a subjectAlternativename:dNSName entry), you may ask us to revoke the certificate by demonstrating your control of the domain.
The portal will list the supported means of demonstrating control and provide straight-forward instructions.
You may choose to revoke one or more certificates by providing the serial numbers of the certificates to be revoked, or to request the revocation of all certificates containing this domain.

The portal will note that #2b discloses the private key to us and means that the key is compromised, and will recommend the use of #2a by preference.

Comment 11

[Ryan Sleevi]

Wayne: Is there any additional information you'd like for this incident report? I think we're at a point of closing it out.

While Comment #10 is not 'ideal' under a perfect world, I think the problems are fairly well-known within this space. I think it may be useful, in future communications, to perhaps survey CAs the methods on how they handle this, or to require the disclosure within the CP/CPS. The point of such an exercise would be to examine the broader industry and look to synthesize good practices into formal requirements. Certainly, the complexity within this space is one of the reasons Google was reticent to enable CT redaction, for example, so I don't think there's a clear or consistent "right" answer yet.

Comment 12

[Wayne Thayer]

Robin: please provide a status update on Sectigo's self-service revocation portal.

Comment 13

[Robin Alden]

(In reply to Wayne Thayer [:wayne] from comment #12)

Robin: please provide a status update on Sectigo's self-service revocation portal.

It is in development now. We anticipate having it QA'd and live around the end of this month (July).

Comment 14

[Robin Alden]

I apologize for the late response. I realize that we let the projected release date go by without providing an update.
This self-service revocation facility is still in development and is now expected to be released on 18-August-2019.

Comment 15

[Robin Alden]

The self-service revocation portal was released on 18-August-2019.
Method 3 on that page, Revocation by proof of domain control, does not yet support wildcard domains but the portal is otherwise functional.
We expect the fix to the wildcard issue for method 3 to be released on 22nd September.

Comment 16

[Wayne Thayer]

Robin: thanks for the update. Despite the "wildcard issue" with the revocation portal. I consider remediation to be complete.