1699756 - Sectigo: Reseller ZeroSSL and Private Key Generation

Status: RESOLVED INVALID

(CA Program :: CA Certificate Compliance, task)

Description

[Ben Wilson]

This issue is being branched off for further discussion from Bugzilla #1698936.

Sectigo reseller ZeroSSL states that Subscriber private keys are generated and stored as follows: https://help.zerossl.com/hc/en-us/articles/360060119993-Security-Insights

Security Insights

November 30, 2020 09:38

At ZeroSSL, security on our platform and on the web, in general, are two of our top priorities. Our team has worked long and hard in order to come up with an SSL certificate workflow that provides both a high level of security, as well as the highest possible level of usability and convenience for our customers.

CSR and Private Keys

One of the reasons creating, renewing, and managing SSL certificates using ZeroSSL is as easy as skipping through a few buttons is a secure system where the certificate key pair (consisting of CSR/public key and private key) is generated on the client-side rather than on the server-side.

Private keys are confidential and can never be stored on any server other than our customer's server in a decrypted format, which is why our system is making use of a unique encryption method, which ensures that functional private keys can only ever be retrieved by our customers when logged in to their ZeroSSL account.

Here is how our encryption method works:

With each login or registration, a user account is assigned a unique encryption key, which is stored locally in the browser. When creating a certificate using ZeroSSL, a key pair is generated, and this very encryption key is used to uniquely encrypt the key pair's confidential private key.

Upon completion of the order process of a certificate, the public key (CSR), as well as the encrypted private key, are stored securely on the ZeroSSL system, waiting to be retrieved whenever the customer requests to download a ZIP containing their certificate files. When a download request comes in, the certificate files, as well as the encrypted private key, are provided by the server to the client (browser). In the next step, the browser retrieves its unique encryption key and uses it to decrypt the encrypted private key returned from the server. Apart from the local browser itself, no other system is capable of performing this decryption.

Finally, a ZIP-file is created automatically by the client (browser) and the certificate download can be processed. The downloadable ZIP-file typically contains two certificate files (certificate.crt and ca_bundle.crt) as well as the functional decrypted private key (private.key).

This process might be in violation of Mozilla Root Store Policy 5.2, https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#52-forbidden-and-required-practices, and this guidance,
https://wiki.mozilla.org/CA/Forbidden_or_Problematic_Practices#Distributing_Generated_Private_Keys_in_PKCS.2312_Files

Comment 1

[Tim Callan]

We don't believe this process can be in violation of Policy 5.2 and the other cited guidance.

This statement from Policy 5.2 does not apply on two counts:
CAs MUST NOT generate the key pairs for end-entity certificates that have an EKU extension containing the KeyPurposeIds id-kp-serverAuth or anyExtendedKeyUsage.

First, ZeroSSL is not a CA. Second, it is not generating keys. There is no discussion in this policy of storage of keys.

This statement from the problematic practices guidance does not apply for the same two reasons:
CAs must never generate the key pairs for TLS server certificates.

Is there a different passage in this guidance you meant to call our attention to that we should be looking at?

Comment 2

[Ben Wilson]

Attachment: Capture2.PNG

Comment 3

[Ben Wilson]

Hi Tim,

I'd like to focus on your second argument, because that is what I'm interested in -- whether ZeroSSL is generating keys. I think you lose the first prong of your argument because ZeroSSL is a reseller of Sectigo, and Sectigo as the operator of the CA cannot absolve itself of responsibility by saying that someone else is doing it, but if necessary, we can review the Baseline Requirements and the Mozilla Root Store Policy for additional things we can focus here in this bug, but for right now I'd like to have additional proof that the described processes are as described. And additionally, as I and Ryan have said, this is about transparency and full and accurate communication, including the reseller's relationship with Sectigo. During my cursory review of the certificate application process, it is clear that Sectigo has integrated its processes with those of ZeroSSL (e.g. performing domain validation, etc.).

Currently, I want to verify that key generation occurs locally and that the private key is encrypted when it is not in the possession of the Subscriber.

I was able to retrieve the Zip file from the ZeroSSL system using different browsers, and there was no password protection on either the Zip file or the private key itself. See attachment https://bugzilla.mozilla.org/attachment.cgi?id=9210434

I have numbered the following ZeroSSL representations for future reference:

1 - "our system ... ensures that functional private keys can only ever be retrieved by our customers when logged in to their ZeroSSL account"

2 - "a user account is assigned a unique encryption key, which is stored locally in the browser."

3 - "When creating a certificate using ZeroSSL, a key pair is generated, and this very encryption key is used to uniquely encrypt the key pair's confidential private key."

4 - "Upon completion of the order process of a certificate, the public key (CSR), as well as the encrypted private key, are stored securely on the ZeroSSL system, waiting to be retrieved whenever the customer requests to download a ZIP containing their certificate files."

5 - "When a download request comes in, the certificate files, as well as the encrypted private key, are provided by the server to the client (browser)."

6 - "In the next step, the browser retrieves its unique encryption key and uses it to decrypt the encrypted private key returned from the server."

7 - "Apart from the local browser itself, no other system is capable of performing this decryption."

8 - "a ZIP-file is created automatically by the client (browser) and the certificate download can be processed"

As I went through the step-by-step process, I also noticed that there appear to have been omissions or inaccurate representations about what was occurring at any given time. For instance, I was never informed if/when a third-party system was causing my local system to generate a private key on my website's behalf.

9 - "Before validation, we will auto-generate contact information and a CSR for your certificate."

10 - "Your certificate has been created and is ready for domain verification." (Pre-validation)

11 - "Congratulations, your SSL certificate is en route! However, you need to verify ownership of your domain before installing your certificate." (Pre-validation)

How can the SSL/TLS certificate have been created and en route if other steps have not yet been performed?

What are Sectigo's and ZeroSSL's plans to improve the security and transparency of the process?

Thanks,

Ben

Comment 4

[David Spitzer-Dulagan]

Hi Ben,

nice to meet you - I'm the CTO of ZeroSSL and this discussion has been brought to my attention today. I hope that I'll be able to give some insight into our "automatic CSR generation" workflow that you seem concerned by - something I can relate to as that feature of our platform was the first I was inquiring about when I joined ZeroSSL.

Even though you already pasted our Helpdesk's official document I'd like to provide a more technical answer on how the process works:

When the automatic CSR generation is selected by the user, both the CSR and the private key are generated using a Javascript library (forge) on the user's client (browser) side. The private key is then encrypted on the client using aes256 with a key that is derived from the user’s plaintext password - this key is generated on login and stored in the browser's local storage and never leaves the browser. The user's plaintext password is never sent to our backend without prior hashing either. The resulting CSR and encrypted private key are then submitted to our backend. The backend saves the encrypted private key on our systems and sends the CSR to Sectigo for signing. Once the backend receives the callback from Sectigo that the certificate has been issued we store the certificate on our backend and the client indicates to the user that the certificate has been issued. The user can then download the certificate using our web portal. Upon clicking on the download link, the client-side javascript retrieves the certificate and corresponding private key from our backend, decrypts the private key again using the same encryption key stored in local storage and generates a ZIP file on-the-fly that contains the certificate, decrypted private key and the ca_bundle files for the user to save locally.

The reason why we store the encrypted private key on our backend is to allow our customers to download their certificates at a later point in time. However, ZeroSSL is never capable of decrypting the private key.

If you don't want to take my word for it, here's a process on how you can verify this (apart from reading the javascript code):

1. Login to ZeroSSL and open Firefox's Storage Inspector developer tool
2. Locate the "encryption_key" (under Local Storage -> app.zerossl.com). This is the key that is being generated on the login form and passed to the aes256 encryption algo and has been derived directly from the password that you entered in the login form input field.
3. Open the Network Inspector and click on the "New Certificate" button
4. Fill out the forms, leaving "Auto-Generate CSR" checked until the "Finalize Your Order" step
5. Upon clicking "Next Step" on the last page the key pairs and CSR are generated and the private key is encrypted (all done on the browser). Afterwards you should see a POST request to https://app.zerossl.com/ajax/advanced_ajax_handler.php?type=zerossl_create_certificate . Please inspect the request body and copy the value of "postArray".
6. You'll find that the POST payload is a json document containing the "certificate_domains", "certificate_validity_days", "certificate_csr" and "certificate_pk_encrypted" fields.
7. Save the json payload to a file named "body.json"
8. Run the following commands to inspect the private key:

# (jq and sed required):
jq -r .certificate_pk_encrypted body.json | sed '/^-----/s/+/ /g' > private_key_encrypted.pem
# On the next command, when prompted, enter the value of the "encryption_key" extracted from the Storage Inspector
openssl rsa -in private_key_encrypted.pem -text

9. The above command should show you the contents of the private key, decrypted with the previously extracted "encryption_key" which was not sent to the server.
10. You will find that when you finish the rest of the process and click on the download link for the ZIP file that the response payload from the server contains - again - the encrypted private key.

Even when inspecting all of the traffic sent back and forth you should see that neither the "encryption_key" nor the user's plain-text password ever leave the browser.

We believe that this process provides the most convenience for our users while being sufficiently secure. It makes a clear distinction between where the keys are generated (only on the user's browser) and the signing process involving our api backends as well as those from Sectigo.

However, if you have any input for us on how to improve this process or still have concerns that it might violate any of Mozilla's guidelines, I'll be happy to work on a solution. Please let me know your thoughts.

Best,
David

Comment 5

[Matthias]

(In reply to Tim Callan from comment #1)

There is no discussion in this policy of storage of keys.

BR s6.1.2 (as of BR v1.3.5, until at least the latest version v1.7.3), which is included by reference, explicitly states "Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key without authorization by the Subscriber.", and after failing to find such authorization mentioned in the T&C of ZeroSSL's website, I believe this indicates that this current scheme does not comply.

(In reply to David Spitzer-Dulagan from comment #4)

The reason why we store the encrypted private key on our backend is to allow our customers to download their certificates at a later point in time. However, ZeroSSL is never capable of decrypting the private key.

I find this statement problematic, because it is only partially correct.

As long as the customer doesn't log in, you are indeed correct. But as soon as the customer does login, they supply their plaintext password into a systems context that you control, which is equivalent to giving your systems access to the private key of that certificate. That you never combine the two anywhere but in the users' browser is nothing but an implementation detail, this could most trivially be changed to send the password or derived decryption key to your servers for remote decryption.

Other than that, I see that the encryption key is stored in localstorage, next to some values from your (what I believe is a) zendesk integration. This to me indicates that this zendesk integration has at least some amount of access to localstorage, and with that, potential access to the decryption key, and with that access to the private keys.

All this indicates that it is very possible for ZeroSSL to get access to the private key, as long as the customer logs in at least once.

Comment 6

[David Spitzer-Dulagan]

(In reply to Matthias from comment #5)

BR s6.1.2 (as of BR v1.3.5, until at least the latest version v1.7.3), which is included by reference, explicitly states "Parties other than the Subscriber SHALL NOT archive the Subscriber Private Key without authorization by the Subscriber.", and after failing to find such authorization mentioned in the T&C of ZeroSSL's website, I believe this indicates that this current scheme does not comply.

Thank you for your reply. We will issue an update to our Terms and Conditions that specifically states that 'When using the "Auto-Generate CSR" feature, ZeroSSL will store the user's encrypted private key on ZeroSSL's servers in order to allow the user to download the private key at a later point in time.'.

Please let me know if you think that further action is required.

Comment 7

[David Spitzer-Dulagan]

We've released a new version of our ZeroSSL Terms and Conditions that explicitly mention the storing of the encrypted private key (under "Accounts, Passwords and Security") as well as the fact that the user is also bound by the Sectigo Certificate Subscriber Agreement. Our sign-up page has been changed to reflect that as well.

Comment 8

[Matthias]

I have revoked the test certificate through your web portal, but in the process I could not find an option to specify 'keyCompromise' as the reason for this revocation. It is my opinion that any key materials that you have stored can be considered compromised when a revocation is requested by the subscriber[0], so I think that this would be a reasonable default when the user requests the revocation of a certificate that has key materials that ZeroSSL has stored at any point in the process.

Additionally (but this is mainly UX) I believe that it would be great if you could add an additional warning that the private key is stored on your systems to the 'Auto-Generate CSR' option, as well as indications in the certificate list for certificates of which you have at any point in time stored the key.

[0] I can't use 2FA on the website, and the password requirements are allright, but not strong (>=8 characters, of which 1 capital, 1 lowercase, 1 number; no password strength/entropy indicator). Even though it might not be directly your problem, account compromise = key compromise for these certificates, and not being able to flag certificates as key compromised where this can reasonably be the case is overall a bad thing for the the ecosystem.

Comment 9

[David Spitzer-Dulagan]

Hi Matthias,

thanks a lot for your input. We're going to release an update in the next few days that'll incorporate the keyCompromise revocation reason as well as a notice directly in the UI educating the user about the functionality and implications of the automatic CSR generation.

Two-Factor authentication is already part of our Q2 roadmap.

Comment 10

[Tim Callan]

ZeroSSL has been very active on this thread, and the ZeroSSL team is the most directly qualified to discuss the technical operation of its service. We are watching this thread with active interest and at this point don't have anything additional to add.

Comment 11

[Ben Wilson]

(In reply to David Spitzer-Dulagan from comment #4)

Hi Ben,

1. Login to ZeroSSL and open Firefox's Storage Inspector developer tool

I did this, but I do not have a "testing" account with free certificates to look into this further. In any event, I think the burden should still be on ZeroSSL to establish that the process is sufficiently secure.

2. Locate the "encryption_key" (under Local Storage -> app.zerossl.com). This is the key that is being generated on the login form and passed to the aes256 encryption algo and has been derived directly from the password that you entered in the login form input field.

When I log in to download an existing zip file, there is no "encryption_key" (under Local Storage -> app.zerossl.com). It says "no data present for selected host." Yet I believe I am able to download an unencrypted zip file containing the private key.

3. Open the Network Inspector and click on the "New Certificate" button

When I click on the "New Certificate" button, I do notice that an encryption key is generated, but as noted above, I do not have an account to conduct extensive testing.

10. You will find that when you finish the rest of the process and click on the download link for the ZIP file that the response payload from the server contains - again - the encrypted private key.

I did not see an encrypted private key, as noted above. In the zip file was the "unencrypted" private key.

Even when inspecting all of the traffic sent back and forth you should see that neither the "encryption_key" nor the user's plain-text password ever leave the browser.

I would need to run these processes again and analyze all traffic back and forth.

Comment 12

[Ben Wilson]

Thanks, David, for leading me through the step-by-step process of the key generation and delivery on the local client machine today. I'm satisfied that it does not present an issue with the exposure of the private key, so I am closing this bug as invalid.