1698936 - Sectigo: ZeroSSL: failure to revoke within 24 hours

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Chris Marget]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Safari/605.1.15

Steps to reproduce:

Attempts to revoke certificates through the ZeroSSL web portal produce an error:

"An error occurred. Please try again or contact support. (Error Reference: certificate cannot be revoked)"

I contacted ZeroSSL support via email, explained that the web portal wasn't working and that the private key for https://crt.sh/?id=4184234902 was compromised[1]. I requested that the certificate be revoked.

8 Mar 2021 23:50:54 +0000
Support request #105410 assigned to the request according to email auto-response.

11 Mar 2021 12:59:40 +0000
ZeroSSL support inquired by email: "Can you please confirm the domain which needs to be revoked?"

11 Mar 2021 15:23:50 +0000
I replied indicating that the certificate in question was https://crt.sh/?id=4184234902

16 Mar 2021 15:07:52 +0000
I replied to the support ticket again, expressing concern that the certificate remained un-revoked:

OCSP Request Data:
Version: 1 (0x0)
Requestor List:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 331FFE3FFD0B8416284F948D56C07E0392D8F64D
Issuer Key Hash: 0F6BE64BCE3947AEF67E901E79F0309192C85FA3
Serial Number: 3687B44BC7A06649E0F98A31DC54A1D0
Request Extensions:
OCSP Nonce:
04100A417B8719528D1C802ED36DC0C6CD83
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: 0F6BE64BCE3947AEF67E901E79F0309192C85FA3
Produced At: Mar 16 09:35:26 2021 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 331FFE3FFD0B8416284F948D56C07E0392D8F64D
Issuer Key Hash: 0F6BE64BCE3947AEF67E901E79F0309192C85FA3
Serial Number: 3687B44BC7A06649E0F98A31DC54A1D0
Cert Status: good
This Update: Mar 16 09:35:26 2021 GMT
Next Update: Mar 23 09:35:26 2021 GMT

Signature Algorithm: ecdsa-with-SHA384
     30:65:02:30:12:bb:97:a0:b2:9f:d5:c8:ce:09:0b:26:00:a7:
     88:8f:40:e8:75:aa:d2:91:ce:f9:85:eb:e8:f1:7d:54:39:eb:
     bc:38:de:14:eb:26:fa:5e:da:ad:57:6f:74:df:e2:ef:02:31:
     00:8c:fd:66:b2:78:03:8a:bb:4a:f6:4d:f0:07:31:1e:40:59:
     75:e6:e8:7a:ba:52:7d:dd:9b:1d:0a:b4:48:89:2b:a4:42:0b:
     e6:fb:50:23:19:f0:40:f8:d1:ac:a0:e9:88

0x3687B44BC7A06649E0F98A31DC54A1D0: good
This Update: Mar 16 09:35:26 2021 GMT
Next Update: Mar 23 09:35:26 2021 GMT

16 Mar 2021 19:19:xx +0000
@sleevi_ tagged @SectigoHQ and @zerosslHQ in a tweet[2] suggesting I file a bug here.

16 Mar 2021 19:28:46 +0000
The certificate was revoked.

[1] https://twitter.com/chrismarget/status/1369065946120388611
[2] https://twitter.com/sleevi_/status/1371888864369999873

Comment 1

[Tim Callan]

This bug needs to be RESOLVED INVALID.

The issuing CA (ZeroSSL ECC Domain Secure Site CA) is governed by our CPS (as shown by https://crt.sh/?id=2427368475), and so one of our Problem Reporting Mechanisms (as defined in our CPS) should have been used. However, in this case, the reporter contacted ZeroSSL only, which means that no Problem Report was actually submitted.

https://crt.sh/?id=4184234902&opt=problemreporting explains how subscribers can order revocation of certificates.

Comment 2

[Chris Marget]

Hi Tim,

The possibility that I was barking up the wrong tree was top of mind throughout this process. In fact, that very question is what led to me being advised to turn here [1].

I understand that consumer education is not the purpose of this bug thread, but I wonder if you'll indulge me with a bit of explanation so that I can better understand how things work and where I went wrong?

1. From my perspective, the process began with an error popup when I clicked the "revoke certificate" button in the ZeroSSL web UI. That error directed me to contact ZeroSSL support. Perhaps I'd gone off the rails by attempting to use the ZeroSSL "revoke" button in the first place[2]. What process are end consumers expected to follow?

2. My (limited) understanding is that revocation is controlled by the private key of the issuing CA, either directly or by delegation (by certificate) to an OCSP responder. Your note suggests that issuance and revocation tasks are handled by different parties. It would never have occurred to me to request revocation from further up the cert chain, because I (wrongly?) believed that the ultimate authority lies with the issuer. Can you help connect these dots for me? What party holds the private key for the CA that issued my certificate?

[1] https://twitter.com/chrismarget/status/1371874526133420033
[2] Well after this adventure began, I found a ZeroSSL page advising that revocation of issued-via-ACME certificates is currently impossible: They can only be revoked via ACME. I found that document via google deep-link, have tried and failed to find using the ZeroSSL admin panel/documentation as a starting point.

Comment 3

[Ryan Sleevi]

Tim: Perhaps instead of suggesting simply closing this, you can detail a bit more about the relationship with Sectigo and ZeroSSL, and how that flow should work.

My understanding is ZeroSSL is a Sectigo reseller, so I imagine this potential issue can come up with a number of other Sectigo resellers. The fact that ZeroSSL has its own revocation portal certainly understandably raises points of confusion; for example, Sectigo's reseller agreement could require that resellers direct to Sectigo's CP/CPS and revocation methods.

I can understand potentially resolving this as WontFix, by suggesting it's not a BR violation, but I do think we should take this opportunity to understand how we can improve the ecosystem, because this does seem a failure to be mindful of.

Comment 4

[Matthias]

Quoting BR s9.6.3:

Prior to the issuance of a Certificate, the CA SHALL obtain, for the express benefit of the CA
and the Certificate Beneficiaries, either:

1. The Applicant’s agreement to the Subscriber Agreement with the CA, or
2. The Applicant’s acknowledgement of the Terms of Use.

Option 1 does not seem to apply here: ZeroSSL's Terms & Conditions [0] do not reference Sectigo as a party in the agreement between the user and ZeroSSL. No mention was found on the ZeroSSL website that the subscriber is bound to a Subscriber Agreement with Sectigo.
Option 2 uses the term "Terms of Use", which is specified in BR s1.6.1:

Terms of Use
Provisions regarding the safekeeping and acceptable uses of a Certificate
issued in accordance with these Requirements when the Applicant/Subscriber is an
Affiliate of the CA or is the CA.

I would argue that the subscriber in this case (if the term may even be applied in this case, due to the lack of contract between this subscriber and Sectigo) is not an affiliate of the CA (ZeroSSL is an affiliate, but not the subscriber), nor is the subscriber the CA (that much is clear), and as such a Terms of Use is not applicable.

I believe that the prerequisites of BR s9.6.3 are not met in this case, and as such I believe that certificates issued to users of ZeroSSL are not issued according to the BR.

[0] https://zerossl.com/terms/

Comment 5

[Ben Wilson]

Attachment: Automatic Key Pair and CSR Generation

I understand from this "Auto-Generate CSR" option at https://app.zerossl.com/certificate/new that ZeroSSL generates a private-public key pair for the subscriber and signs the CSR and makes a zip file available containing the private key and certificate. How does this comply with Mozilla Root Store Policy section 5.2, which says, "CAs MUST NOT generate the key pairs for end-entity certificates that have an EKU extension containing the KeyPurposeIds id-kp-serverAuth or anyExtendedKeyUsage."?

Comment 6

[Matthias]

Note that it is also a BR requirement, in s6.1.1.3:

If the Subscriber Certificate will contain an extKeyUsage extension containing either the values id-kp-serverAuth [RFC5280] or anyExtendedKeyUsage [RFC5280], the CA SHALL NOT generate a Key Pair on behalf of a Subscriber, and SHALL NOT accept a certificate request using a Key Pair previously generated by the CA.

Which contains the suprising constraint (when taken literally) that CAs may not sign their own server certificates as long as they themselves generated the key described in that certificate. (https://github.com/cabforum/servercert/issues/255)

Comment 7

[Jean Charette]

Attachment: Generate CSR (MrDomain).png

Related to #com(In reply to Ben Wilson from comment #5)

I understand from this "Auto-Generate CSR" option at https://app.zerossl.com/certificate/new that ZeroSSL generates a private-public key pair for the subscriber and signs the CSR and makes a zip file available containing the private key and certificate. How does this comply with Mozilla Root Store Policy section 5.2, which says, "CAs MUST NOT generate the key pairs for end-entity certificates that have an EKU extension containing the KeyPurposeIds id-kp-serverAuth or anyExtendedKeyUsage."?

I have seen something similar being done by MrDomain (which I think is also a Sectigo reseller) https://www.mrdomain.com/

Comment 8

[Ben Wilson]

For recordkeeping and discussion purposes, the issue discussed in Comments 5-7 should be opened as its own incident. I have created Bugzilla Bug #1699756 for such purpose.

Comment 9

[Ben Wilson]

I agree with Ryan's Comment #3, we need to have more transparency on this reseller relationship and clearer communication on reseller websites on who is the CA, which CPS applies, and how a subscriber or third party can request revocation, among possibly other items I haven't thought of yet.

Comment 10

[Tim Callan]

(In reply to Matthias from comment #4)

This is not a BR violation. This certificate was ordered by ACME, and before using ACME, the subscriber had to agree to our Terms of Service at secure.trust.provider.com/repository/docs/Legacy/20201020_Certificate_Subscriber_Agreement_v_2_4_click.pdf as demonstrated by ZeroSSL’s directory URL: https://acme.zerossl.com/v2/DV90

Comment 11

[Tim Callan]

(In reply to Chris Marget from comment #2)
To clarify, it is always possible to revoke a certificate by contacting Sectigo directly or by ACME using the private key.

Comment 12

[Ben Wilson]

Hi Ryan,
Should we close this bug as Invalid and move ongoing discussion of comment #3 to Bug #1699756 ?
Thanks,
Ben

Comment 13

[Ryan Sleevi]

Hey Ben,

I think I'd still like to see a discussion of Comment #3 on this. I don't think Sectigo has really responded here (unless they're assuming Comment #10 is the fullness of the reply), and Bug 1699756 seems somewhat separate to my questions in Comment #3.

Comment 14

[Matthias]

This is not a BR violation. This certificate was ordered by ACME, and before using ACME, the subscriber had to agree to our Terms of Service at secure.trust-provider.com/repository/docs/Legacy/20201020_Certificate_Subscriber_Agreement_v_2_4_click.pdf as demonstrated by ZeroSSL’s directory URL: https://acme.zerossl.com/v2/DV90

Yes, indeed, sorry for the noise on that part.

But, there are still issues through the web portal flow: I just had https://crt.sh/?id=4256123157 issued for one of my test domains through their web portal, but in none of the steps that I went through did I receive an indication that I went into a legal agreement with Sectigo / would be bound to the terms of service of Sectigo; only those of ZeroSSL.

Comment 15

[Julian Zehetmayr]

Hi Matthias,

This is Julian from ZeroSSL. Just a quick heads-up that we're working on a solution here to include Sectigo's TOS within our web portal process, and we'll update here when it's done, likely in the next 24-48 hours.

Thanks!
Julian

Comment 16

[David Spitzer-Dulagan]

Hello everyone,

we've released a new version of our ZeroSSL Terms and Conditions that explicitly mention the storing of the encrypted private key (under "Accounts, Passwords and Security") as well as the fact that the user is also bound by the Sectigo Certificate Subscriber Agreement. Our sign-up page has been changed to reflect that as well.

Comment 17

[Tim Callan]

(In reply to Ryan Sleevi from comment #3)
The ZeroSSL revocation portal does not work with certificates issued through ACME. ZeroSSL has modified its language to clarify this point. For example, its knowledgebase article states:
"Currently, certificates issued via ACME can not be revoked from inside the portal – please follow the instructions of your ACME client for revoking those certificates."

We understand the previous omission of such explanation to be an oversight owing to the different requirements for customers using ACME from those using the ZeroSSL interface. ZeroSSL is unique among Sectigo resellers in its decision to offer direct ACME control of certificates to its end customers, so this partner is breaking new ground and discovering best practices as it goes.

It's worth a brief aside to explain why a reseller portal cannot revoke a certificate ordered through ACME. https://tools.ietf.org/html/rfc8555#section-7.6 says:
"Revocation requests are different from other ACME requests in that they can be signed with either an account key pair or the key pair in the certificate... Before revoking a certificate, the server MUST verify that the key used to sign the request is authorized to revoke the certificate."
Revocation requests via non-ACME channels are not suitably signed.

In cases other than ACME, we do allow the partner who originally ordered the certificate to send us a revocation request, which we will honor. And of course we always will honor a direct revocation request from the certificate holder as well. That maximizes ease of revocation for subscribers.

To drill down further, we have to segment our partner base. First, we have a very large segment of hosting providers. Hosting partners manage the physical machines and the certificates they contain. In this case the relationship is very clean in that the customers rarely know anything about servers, certificates, or keys. Partners can send revocation and replacement requests directly by API, and they deal with installation of new certificates.

The second segment is what we’ll call a “pure reseller.” These are partners that own the customer interaction due to their focus on a specific segment, existing customer relationships, business model innovation, or direct marketing and SEO savvy. Their SSL business is fundamentally a retail channel. ZeroSSL falls into this category. These partners may offer some kind of online account and management portal in order to provide better service, and that can but doesn’t need to include the option to revoke or to revoke and replace. As stated above, we enable this service by honoring revocation requests from our resellers for certs they sold. These customers, of course, also can come directly to us with revocation requests. As they are likely to be technically savvy, this is what they most often do.

Then there is an enterprise sales channel, which once upon a time we might have referred to as VARs or SIs. These resellers are bringing in and setting up PKI and certificate management solutions and then letting a sophisticated IT staff manage them. In this scenario revocation requests are highly likely to come straight to Sectigo.

For the first segment, there is no concern as the partner handles these matters. For the second segment, only the combination of ACME service with the presence of a customer revocation portal – unique to ZeroSSL – enabled this problem to occur. The third segment is populated by IT professionals who are responsible for knowing how to handle certificate lifecycle needs. That this is a corner case is also testified to by the fact that nobody around here can think of another example of this particular problem. Subscribers who can’t figure out how to revoke their certs simply haven’t come up before.

With regard to Subscriber Agreements, resellers are required contractually to provide our Subscriber agreement and obtain consent. Our standard reseller agreement states in part,
“In order to receive the Subscription Services, each Subscriber must execute, electronically or in writing, a Subscriber Agreement applicable to the ordered Subscription Services. Such Subscriber Agreement is between Sectigo and the Subscriber only. Reseller shall not be considered a party to the Subscriber Agreement.”

In the event that we learn of a breakdown in this process, of course we work with the reseller to help fix it. Again, this is exceedingly rare. There is no desire among our partner base to prevent an executed Subscriber Agreement, so any such occurrence is simply error or oversight.

Comment 18

[Tim Callan]

I intended my comment #17 to "detail a bit more about the relationship with Sectigo and ZeroSSL, and how that flow should work" and believe I did that. Are there additional questions from the community on what I wrote in that comment?

Comment 19

[Ben Wilson]

I will schedule this to be closed next Wed. 14-Apr-2021, unless there are any other matters to discuss.