1896462 - Digicert: Preview certificate uploaded to CCADB instead of the actual certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Jeremy Rowley]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

Steps to reproduce:

SUMMARY:

Before each key ceremony for a publicly trusted ICA, our key ceremony tool generates a “preview” certificate. This certificate is identical to the final certificate except the preview certificate is not signed by a trusted root and has a different serial number. This preview certificate is linted and reviewed prior to signing the actual ICA.

DigiCert has a legacy tool for root management called “Rootica”. We are in the process of replacing this tool with an updated tool called “PrimaRoot”.

When testing the CCADB API upload function, we found 1 preview certificate that was uploaded to CCADB from the staging environment instead of the publicly trusted certificate.

Rob Stradling, Sectigo, noticed the error in crt.sh and sent a personal email to a Digicert employee. However, the email went to a junk folder and was not noticed in time to correct the error.

After realizing the error, we uploaded and revoked the trusted ICA. This ICA was not used to issue any end-entity certificates and was not in our production environment yet.

IMPACT

TIMELINE -

All times are in GMT

17 April 2024 19:35 – Primaroot deployed

24 April 2024 18:02 - Preview ICA created.

24 April 2024 19:00 – Publicly-trusted ICA signed.

24 April 2024 21:13 - Preview cert automatically uploaded to PrimaRoot stage.

24 April 2024 22:08 - Preview Cert automatically uploaded to CCADB via API from PrimaRoot Stage

30 April 2024 at 13:15 - Email sent from Sectigo about the upload.

10 May 2024 11:57 - Email found, and Digicert started investigating.

10 May 2024 13:33 – Correct certificate uploaded to PrimaRoot production and CCADB.

10 May 2024 16:36 – Fix deployed to check signature of uploaded certificate.

10 May 2024 21:45 - Certificate revoked and marked in CCADB

ROOT CAUSE ANALYSIS

Neither DigiCert nor CCADB had signature checking in place before uploading an ICA

The preview certificate was uploaded to the stage environment instead of the certifiate signed by ‘DigiCert Assured ID Root G2’.

The process for uploading certs to CCADB from PrimaRoot involves generating a hash using the Issuing Subject Distinguished Name and matching this hash against the PrimaRoot database to locate the parent's pem file. These parameters are then utilized to construct the API payload sent to CCADB. A critical flaw with this approach is that dummy certs possess identical Issuing DNs as the genuine certificates. This let the dummy ICA upload instead of the acutal certificate as both were saved to stage.

We normally do not save preview certificates to stage. However, we were testing Primaroot’s ability to automtically upload in stage, which is why the preview certificate ended up on stage.

LESSONS LEARNED

WHAT WENT WELL

New system worked as expected to automatically upload certificates.

We uploaded two publicly trusted ICAs at the same time and one ICA was the correct version.

WHAT DIDN'T GO WELL

Because the preview certificate and trusted certificates looked identical in the tool, the manual review failed to notice that the preview certificate was uploaded in place of the trusted certificate.

WHERE WE GOT LUCKY

The ICA was not in production so no end entity certificates were impacted.

DigiCert investigated, revoked, and uploaded the revoked certificates within 24 hours of detecting the issue.

ACTION ITEMS

PrimaRoot now performs signature checking before uploading new certificates to CCADB. This ensures that only the real certificate is uploaded instead of the preview certificate. | Preventative| Done

Certificates are uploaded only from production and not stage. | Preventative| Done

APPENDIX
DETAILS OF AFFECTED CERTIFICATES
One impacted ICA:

trusted: https://crt.sh/?sha256=A9A0073F6BBA1509995DAF51991CCB1337C0DAA65099F9C1D108AA520285A7C5

dummy: https://crt.sh/?sha256=21B72E7753C711B3027592C4E5CDF7D8EBF374224E6BC03EDC217DBF39CFA081

Comment 1

[Rob Stradling]

Rob Stradling, Sectigo, noticed the error in crt.sh and sent a personal email to a Digicert employee. However, the email went to a junk folder and was not noticed in time to correct the error.

Just to explain why I did that: From the evidence available to me, it wasn't clear that there was a "problem" in BR terms. So it seemed appropriate to send a less formal notification that wouldn't set a deadline for a response.

These parameters are then utilized to construct the API payload sent to CCADB. A critical flaw with this approach is that dummy certs possess identical Issuing DNs as the genuine certificates. This let the dummy ICA upload instead of the acutal certificate as both were saved to stage.

I find it surprising that CCADB accepted this submission. When a new Subordinate CA Certificate is submitted, does CCADB not verify its signature? Does the CCADB web interface behave the same way as the CCADB API in this regard?

trusted: https://crt.sh/?sha256=A9A0073F6BBA1509995DAF51991CCB1337C0DAA65099F9C1D108AA520285A7C5

This link didn't work. I've just grabbed the cert from the CCADB CSV report and submitted it to the Sectigo Dodo log, so crt.sh should ingest it shortly.

Comment 2

[Ben Wilson]

(In reply to Rob Stradling from comment #1)

I find it surprising that CCADB accepted this submission. When a new Subordinate CA Certificate is submitted, does CCADB not verify its signature? Does the CCADB web interface behave the same way as the CCADB API in this regard?

I agree, we should look into these two questions and create a bug in the Common CA Database component.

I have created Bug #1896487 to address this issue.

Comment 3

[Rob Stradling]

I've just grabbed the cert from the CCADB CSV report and submitted it to the Sectigo Dodo log, so crt.sh should ingest it shortly.

Ah. I just discovered that crt.sh's CT log ingestion had fallen behind over the last few days. Not sure why that happened, but it seems to be speeding along OK now. However, it may take a few days to catch up on the backlog.

Comment 4

[Ben Wilson]

So, just to clarify, because of the issue described in Comment #0, DigiCert missed the 7-day reporting deadline for the true subordinate CA to the CCADB (Serial No. 086cb42b1ef4ccd24a0df305d63cecda), and then revoked it. In looking at the CRL, Serial No. 086cb42b1ef4ccd24a0df305d63cecda has been revoked, but serial # 07610352DF363D196508D34C8852139F8FB1C86B has not been revoked (i.e. because it was never issued/signed by the DigiCert Assured ID Root G2). I would think that it would be appropriate to delete the CCADB record for the test CA certificate, because it is not supposed to be in the CCADB.

Comment 5

[Jeremy Rowley]

Hey Ben - what else do you want from us on this bug? We added signature checking on our side and revoked the ICA. Can we close this bug?

Comment 6

[Ben Wilson]

I will close this next week (May 27-31) to allow questions if there are any.

Comment 7

[Ben Wilson]

I'm closing this. I'll also remove the preview certificate from the CCADB, since it was not signed within a trusted-root hierarchy.