Here is our incident report:
1 – How and when we became aware of the problem.
At approximately 14:09 MST on Sunday, January 27, 2019, we read the above-referenced email from Corey Bonnell sent to the mozilla.dev.security.policy list and learned that the OCSP responses for nine CA certificates listed above were erroneous.
2 a. -- Timeline of actions we took in response (unless otherwise indicated, all times are MST Sunday, January 27, 2019).
14:09 – 16:04 We immediately responded to Corey Bonnell’s email and started to review the nine CA certificates, including their AIAs. We determined that they all chained up to the Baltimore Cybertrust Root and pointed to OCSP responses from http://ocsp.omniroot.com/baltimoreroot, except for last one (Serial No. 0c:d1:ea:4d:07:7d:40:2f:89:9c:cc:ba:01:6c:a1:92), which pointed to http://ocsp.digicert.com. We continued our review of the certificate status, reviewed the CRL, and began internal communications with our PKI Operations team to determine how best to correct the erroneous OCSP responses.
16:04 – 16:45 We logged into the OCSP responder and began to pull up CA certificates by serial number and updated their status to revoked. We completed the first update of the certificate records and pushed out OCSP responses to content distribution networks at 16:45.
16:45 – 17:48 A comparison of the OCSP responses with the CRL indicated that we still needed to provide original dates and times for revocation. We conducted a second run through the OCSP database to specify revocation dates and times and we pushed out another set of OCSP responses to content distribution networks.
17:48 We emailed Corey Bonnell and the MDSP list and indicated that our review of the OCSP responses confirmed what he had found and that our updates to OCSP responses were complete and provided the correct “revoked” responses.
2 b. -- Timeline of events before incident was reported
2015 – DigiCert acquired the Baltimore Cybertrust Root from Verizon. We took possession of cryptographic hardware and the private key, but the domains and URLs for CDP and AIA remained with Verizon, so DigiCert would push updated CRLs to Verizon for publication at its CDP. Verizon’s OCSP responder provided responses based on the CRL.
2016-2017 – OCSP responses continued to be provided by Verizon’s OCSP responder. The parties began discussions on revising DNS entries with CNAME redirects so that DigiCert could publish CRLs and OCSP responses directly to the Internet.
2018 – The parties continued discussing the redirection of CDPs and AIA URIs for OCSP and the logistics of cutting over from Verizon’s CDP and OCSP responder to DigiCert’s CRL distribution and OCSP Responder. In preparation for the cutover, we populated a database with the PEM files of unexpired (good and revoked) CA certificates issued by the Baltimore Cybertrust Root. At some point in the process, the revoked flag for some of the certificates was lost and not fixed even though we conducted pre-cutover testing of OCSP responses using the openssl command “ocsp -issuer [Baltimore.pem] -serial 0x…, etc. We were not able to identify the exact reason why these flags were not fixed other than possibly human error. Nevertheless, on June 14, 2018, we initiated the DNS cutover and transitioned to DigiCert’s OCSP responder and did not recheck the status of OCSP responses thereafter.
- Whether the problem has been resolved
Once we were aware of the problem on 27-Jan-2019, we corrected the problem and OCSP responses were correct within 4 hours.
- Summary of the Problem
The problem arose because for the Baltimore Cybertrust Root we engaged in a manual process to work around our inability to publish OCSP responses directly. It was then perpetuated because we did not recheck for the correct OCSP responses following the cutover to DigiCert’s OCSP.
- Complete Certificate Data
As noted above:
crt.sh URL(s), notBefore, notAfter, subject CN, issuer CN
https://crt.sh/?id=3528065 , 2014-02-12, 2021-02-12, Bechtel External Policy CA 1,Baltimore CyberTrust Root
https://crt.sh/?id=91478106 , 2014-04-16, 2024-04-16, Dell Inc. Enterprise CA,Baltimore CyberTrust Root
https://crt.sh/?id=12625621 , 2014-04-16, 2024-04-16, Dell Inc. Enterprise CA,Baltimore CyberTrust Root
https://crt.sh/?id=91478107 , 2014-04-16, 2024-04-16, Dell Inc. Enterprise CA,Baltimore CyberTrust Root
https://crt.sh/?id=12620974 , 2014-09-10, 2024-09-10, Dell Inc. Enterprise CA,Baltimore CyberTrust Root
https://crt.sh/?id=6906659 , 2015-03-03, 2022-03-03, ABB Intermediate CA 3,Baltimore CyberTrust Root
https://crt.sh/?id=6976985 , 2015-03-18, 2022-03-18, Bechtel External Policy CA 1,Baltimore CyberTrust Root
https://crt.sh/?id=35335507 , 2015-05-21, 2022-05-21, ABB Intermediate CA 3,Baltimore CyberTrust Root
https://crt.sh/?id=78292184 , 2016-11-30, 2020-11-30, Eurida Primary CA,Baltimore CyberTrust Root
Other revoked CA certificates under the Baltimore Cybertrust Root may have generated “good” OCSP responses, but those CA certificates did not contain any AIA for OCSP. Any problem with those was fixed in conjunction with the fixes mentioned in this incident report.
- How and why this happened and how it went undetected until now
After going live with the OCSP change back on June 14, 2018, we should have run another OCSP response comparison with the CRL to confirm that all certificates in the database had been appropriately marked as valid or revoked according to the CRL. Because we didn’t run any other post-live queries, this problem went undetected until the advisory from Corey Bonnell. Also, most of these CAs had ceased issuing certificates. Also, very few of the CA certificates issued by the Baltimore Cybertrust Root had an OCSP AIA, so we did not focus sufficiently on those that did.
- List of steps being taken to prevent this in the future
We have changed the way we do revocations for the Baltimore Root and have integrated revocation and OCSP updates for that root into the existing process that we use for other CAs—before the process was manual, now it is programmatic. Specifically, when the CRL is updated the system will update the OCSP responder database as well. As an added precaution, following CRL generation for the Baltimore Cybertrust root, we will also run an openSSL OCSP check on the revoked serial number. Also, in the future when DigiCert acquires a new CA, we will include provisions in the contract that ensures we obtain clearer access rights to the necessary portions of the domain so that we are able to publish CRLs and OCSP responses according to the CDPs and AIA URLs in the certificates according to which the acquired CA has issued.