1653475 - DigiCert: Key Size Not Divisible By 8

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36

Steps to reproduce:

We saw this bug for DFN (https://bugzilla.mozilla.org/show_bug.cgi?id=1651132) and decided to conduct our own search using our compliance analytics team. We found roughly 290 certs issued by various CAs with bad key sizes that have not been revoked. I'm providing notice to all the non-DigiCert ones at the same time as this bug posting.

DigiCert had 24 of the 290, 22 of which are already revoked. The last two were discovered today by scanning our internal CA and will be revoked within the five day timeline. All of these are older certificates that were missed during the previous scan for this issue.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

After reading thought the bug posted by DFN-PKI https://bugzilla.mozilla.org/show_bug.cgi?id=1651132 DigiCert had their compliance analytics team prioritize a scan for RSA keys that were not divisible by 8. We scanned all non-expired certificates. After detecting an issue, we ran a scan on the CA to determine if there were additional issues or non-logged certificates.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

12-Feb-2019 - Code blocking RSA keys not divisible by 8 implemented
12-Feb-2019 - Scan run over all issued certs to determine if the key is divisible by 8
7-July-2020 - Bug posted on Bugzilla by DFN-PKI https://bugzilla.mozilla.org/show_bug.cgi?id=1651132
9-July-2020 - Bug was picked up and Analytics team tasks to Prioritize checks to confirm compliance.
13-July-2020 - Reports for the Entire Cert population were generated finding 192 incorrect Leaf certificates.
14-July-2020 - Confirmed from that list 22 certificates belonging to DigiCert found and that the data was correct.
13-July-2020 - Full examination of CA to ensure issue is blocked at CA. Confirmed blocked at CA. At this point we were having trouble with Censys and the data it was showing. It took a bit to confirm the results.
13-July-2020 Certificates scheduled for revoke and notifications were sent out to customers.
14-July-2020 - Review of previous scan for certs issued. Although the scan results are available, we believe there was an error in the SQL query back in 2019 that was part of scan of our CA.
15-July-2020 - 22 Certificates revoked.
16-July 2020 - Run a full scan on CA to confirm the findings of analytics team and find additional certs. Found two additional certs and confirmed certificate list as bag. Scheduled revocation of these two remaining certificates.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Certificates with a bad modulus are were hard blocked at the CA starting in Feb 2019. This block was implemented when addressing the 521 issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1518555). All certificates in scope of this bug were issued prior to this patching, so no new signings have or will occur. We also have zlint checking all issuance requests. Finally, we have a code review process in place that is substantially different from the process in 2019 (thanks primarily to a change in how we do code pushes after https://bugzilla.mozilla.org/show_bug.cgi?id=1595921). Given all of these changes, we won't have a similar issue.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Crt.sh Serial
https://crt.sh/?id=742789992 01e74b7b34fce1d43f2a695def11d903
https://crt.sh/?id=445345605 098e85ec2b137e4e3b7d4c6ef4d45dca
https://crt.sh/?id=775417637 0d0ec3719f86da395b9cad8dcbef02a5
https://crt.sh/?id=628119486 09673ccc57604d739841cfe9f13c8229
https://crt.sh/?id=637898653 0d14c976f1989d15dd4b210ff0858268
https://crt.sh/?id=803817735 071293a72ad3638752ae14ea5c490a6c
https://crt.sh/?id=742376641 0366aa6168d6a98e6e83ac2b5b06139d
https://crt.sh/?id=435700322 0cdd6b4eb8205bf630c525b5e700c22d
https://crt.sh/?id=620417418 0cf76f4172aae4862b5b7fc00f9756b8
https://crt.sh/?id=796270574 049902a11a2833241c766639ba71f46d
https://crt.sh/?id=185739636 292b7aeb95475244a437adffa4cb4732
https://crt.sh/?id=704072454 0f3823cf8bd944c5d68fcaac5b0e16f8
https://crt.sh/?id=166464104 35928e67371fb5cea15b64a823caaf64
https://crt.sh/?id=1862939195 0b01631153a64c62daa2f27ceda2f3ec
https://crt.sh/?id=255353050 6517df613f8f660d8b390dc086376ce3
https://crt.sh/?id=611796196 0a2963afce537f393acee176f72e06ee
https://crt.sh/?id=620033350 054fb74d624139fc3e8049ffee486f0e
https://crt.sh/?id=627550183 04b639bc08a3953bbb05b21a30a57364
https://crt.sh/?id=706808493 0e42a2d554bd4365f0589ce18d0d6e14
https://crt.sh/?id=274350300 0e8113a7f60b2ee4ff01b8d7ee4cc8bf
https://crt.sh/?id=329501277 0eb1b391151e5a54b43a217373a8fa44
https://crt.sh/?id=738412955 0cf62a3d37ba72a5cc6e38df046d2739
No crt.sh yet (logged but waiting for the next publication)
039E0076DFFF34EA5CBDEFFA740364A0
0254525FC2C494D2F45AAE034D233E15

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   See #4 above.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We added a block for bad key sizes in connection with the 521 issue (mentioned above) back in Feb 2019. We performed a scan of the CA at that time but the scan failed to detect these certificates, due to an issue in the query. We've since changed our dev process, hired an analytics team, and are watching all bugs filed for all CAs, using those issues as both a learning opportunity and as a way to prioritize research by our analytics team. The analytics team is also combing its way through both the BRs and Mozilla policy to detect issues, both past and present.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As part of DigiCert’s ongoing compliance process, we are scanning all data to find issues. The analytics team is using Censys to find issues and then correlate them with our internal database of certs.

This particular issue was patched already to stop issuance. We do not expect this to occur again due to this, but will continue to monitor Censys to detect issues.

Comment 1

[Jeremy Rowley]

The last two are logged to CCADB and revoked. Anything you'd like us to do with this bug before closing it?

Comment 2

[Ben Wilson]

I will schedule this bug for closure on or about 28-July-2020 unless further issues or questions are raised.

Comment 3

[Ryan Sleevi]

I'm a little nervous about the light treatment of the issue ("due to an issue in the query"), given that DigiCert has had a number of issues related to this in the past. I appreciate the "going forward", but I also hope DigiCert carefully examines its past incidents (e.g. Bug 1526154) as part of building a holistic understanding. For example, that change was implemented 2019-02-12, but DigiCert was already aware of the need to examine past issuance in 2019-02-01. Understanding why those controls didn't historically catch the issue are, I think, relevant to understanding the controls going forward.

It may be that we close this as-is, which I think is OK, but I think it'd be crucial, given the pattern, DigiCert is confident they will never miss something again.