1744795 - DigiCert: Issuance of certs with weak keys (ROCA)

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36

Steps to reproduce:

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Hanno Bock reported six certificates that were issued with keys impacted by ROCA (Return of the Coopersmith Attack) on December 2, 2021. He reported an additional three certificates on December 3, 2021. We kicked off our own investigation and found an additional three certificates.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

December 2, 2021: DigiCert receives an email through our certificate problem report alias with six certificates that are impacted by ROCA issues. Digicert confirms certificate issues. DigiCert works on setting up a scanning tool to check whether additional certificates were issued with problematic key pairs.
December 3, 2021: Revocation of the six reported certificates.
December 3, 2021: DigiCert receives a certificate problem report with an additional three certificates. DigiCert confirms issues and starts revocation process. DigiCert launces internal scanner to detect whether any other certificates are impacted
December 3, 2021: One additional certificate is discovered by DigiCert staff
December 4, 2021: Additional certificates revoked.
December 7, 2021: Scan completes of all certificates. No additional certificates with ROCA detected. Additional scans for other vulnerabilities commenced. Code complete on ROCA fixed. Release is pending QA.

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

DigiCert has a check for weak keys but that check is primarily for detection of Debian keys. We are expanding this detection to include ROCA and other weak keys. The tool for ROCA is currently in QA and will deploy after QA is complete, likely later this week or next week. We are also working on a fix for the close primes vulnerability but do not have an ETA on the fix.

4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

We found 12 certificates with detected weak keys. The first certificate issued on September 5, 2021, and the last certificate issued onto October 29, 2021.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

ROCA:
https://crt.sh/?id=5488529785
https://crt.sh/?id=5488520203
https://crt.sh/?id=5164526520
https://crt.sh/?id=5164526469
https://crt.sh/?id=5164529894
https://crt.sh/?id=5164529862
https://crt.sh/?id=5164529897
https://crt.sh/?id=5164530097
https://crt.sh/?id=5164533905
https://crt.sh/?id=5164530051

Close Primes:
https://crt.sh/?id=5503233136
https://crt.sh/?id=5503138827

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

DigiCert did not have a pre-issuance check for the ROCA affected keys.

ROCA information:
CVE-2017-15361
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15361

ROCA: Vulnerable RSA generation (CVE-2017-15361)
https://crocs.fi.muni.cz/public/papers/rsa_ccs17

Revocation occurred within 24 hours of the report per BR 4.9.1.1.:
2. There is clear evidence that the specific method used to generate the Private Key was flawed;
3. The CA is aware of a demonstrated or proven method that exposes the Applicant’s Private Key to compromise;
5. The CA is aware of a demonstrated or proven method to easily compute the Applicant’s Private Key based on the Public Key (such as a Debian weak key, see https://wiki.debian.org/SSLkeys).

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

DigiCert is implementing a check for weak keys beyond Debian weak keys. These checks include ROCA and close primes. The check will reject any enrollments with keys affected by a detected vulnerability. The implementation of the block for ROCA is expected later this week or early next week. We are looking at a third party tool (https://github.com/Ganapati/RsaCtfTool) to see what other checks are necessary to prevent issuance using weak keys.

We are also scanning for all weak keys to make sure there are no other certificates affected by this vulnerability. The scan for ROCA is complete. We are currently scanning for close primes.

Comment 1

[Rob Stradling]

Hi Jeremy. Previously, in response to a question about "what has been done to prevent new ones from being issued" you wrote:

We installed a patch to stop accepting ROCA keys for TLS certs on 2017-10-26.

And yet in comment 0 you wrote:

DigiCert did not have a pre-issuance check for the ROCA affected keys.

I'm struggling to reconcile these two statements. Are you able to explain?

Comment 2

[Jeremy Rowley]

Thanks Rob. I've been combing through our internal history, and I have an answer. In 2017, we acquired the Symantec business. Symantec had a compliance checker called "Sentinel". This tool was essentially a pre-zlint linting tool with various scans. When we added the ROCA check, we added it to this Sentinel system as a pre-issuance check. We also implemented the check on the Symantec email cert system called MPKI8. This is the reference to email certs coming shortly after that I made on the 2017 thread. The ROCA check is still there on MPKI8.

In 2019, we shut down all of the legacy Symantec TLS systems. At that time, we also shut down the Sentinel system. Because Sentinel was used to check ROCA, we turned off the ROCA check with that system shut down. DigiCert was using (and still uses) its own cert linter which is pretty much zlint plus a couple of checks. This system was a separate check from Sentinel. A lot of the checks were redundant and so I didn't think shutting down Sentinel would cause an issue.

In fact, before shutting down zlint, we did an analysis on which checks existed in both systems. The checks where there wasn't a 100% parity included:

Has enough SCTs (handled at issuance by DC systems)
Has diverse SCTs (handled at issuance by DC systems)
Prohibited subject fields in DV (handled by the cert profile)
Wildcard in EV prohibited (handled by the cert profile)
Has Unique Serial Number (handled by database constraints)
Has valid AKI (handled by the cert profile)
Embargo Country / TLD check (handled by a separate system for trade compliance)
Blacklisted Public Key Checks (handled by a system called Goblin)
Compromised Keys (handled by Goblin)
Debian Weak Keys (Added to the DC check)
Heartbleed (Added to the DC check)
ROCA public key check - not added

Embarrassingly, I signed off on the shut down while ROCA was still a gap between DigiCert's linting system and Sentinel. So the check was never implemented as part of the DC linting system. The ROCA check was implemented on the legacy Symantec linting tool and was shut down with the rest of the legacy Symantec TLS systems.

Comment 3

[Jeremy Rowley]

For clarity, the heartbleed check was added to our server inspector software, not to the linting tool. It's a separate service from the pre-issuance checks.

Comment 4

[Jeremy Rowley]

Yesterday, we deployed a check at the CA to evaluate whether a key is impacted by ROCA or close primes. The check is a hard block on detected keys. We're still looking at other checks that are feasible to implement at the CA level.

Comment 5

[Jeremy Rowley]

We've deployed ROCA, small prime, and close prime detection and blocks on all but our smime system. We are still working on adding the service to that system. I'm hopeful we will have it deployed in the next two weeks.

Comment 6

[Jeremy Rowley]

We deployed the same checks to our sMIME system yesterday. I think this bug is remediated now and, unless there are additional questions, is ready to close.

Comment 7

[Ben Wilson]

I'll schedule this to be closed on or about Friday, 14-Jan-2022, unless further discussion is needed.