1820269 - DigiCert: 4 CRLs unavailable or not responding

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martin Sullivan]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were monitoring SSLMate’s CRL-Watch website when we noticed that DigiCert had 4 CRLs listed (see below) with 404 error.

http://kr.crl.digicert.com/DigiCertSecureSiteKoreaEVCA.crl
http://kr.crl.digicert.com/DigiCertSecureSiteKoreaCA.crl
http://kr.crl.digicert.com/DigiCertSecureSiteKoreaECCCA.crl
http://kr.crl.digicert.com/DigiCertSecureSiteKoreaEVCA.crl

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times are MST
1st of March 2023 14:57 DigiCert personal discovered a problem with 4 of our CRLs.
1st of March 2023 16:14 Our own internal alerting system that monitors SSLMate’s CRL-Watch website notified us of the problem.
1st of March 2023 16:16 Compliance confirmed the finding and escalated to Engineering.
1st of March 2023 17:37 Engineering resolved the misconfiguration and pushed out the change to our Content Delivery Network (CDN).
1st of March 2023 17:51 Engineering started receiving the correct response from the effected domain.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This is impacting CRL responses only with no issues with certificates.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

These ICA’s have no current valid nor revoked certificates.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   

There are no impacted certificates since these are CRL response issues only.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The problem was due to a misconfigured path in the CDN configuration. We had recently migrated clusters within our CDN and the origin path for the effected domain (kr.crl.digicert.com) was misconfigured. Our Engineering team had to correct this and push out the changes. We then continued to manually monitor it for a few hours after.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We will effectively manage changes and detect potentially any misconfigurations by adding change management auditing to all our CDN configurations. Going forward we will have a separate person review the changes before deploying them.

As additional improvements to our monitoring, we will add monitoring to our complete list of CRL distribution endpoints disclosed in CCADB rather than a sample set, being this is a large amount of CRL’s to add we expect this to be done at latest by end of Q2.

Comment 1

[Martin Sullivan]

Whilst investigating this issue we have found some other other irregularities linked to this change, we plan to have a revised bug report next week.

Comment 2

[Ryan Dickson]

Hi Martin,

In your subsequent update, can you please add specificity to better describe what you mean by "effectively manage changes" and "adding change management auditing"? Additionally, can you please share more specificity regarding the anticipated time these changes will be made (as you did for the monitoring commitment).

Thanks,
Ryan

Comment 3

[Martin Sullivan]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were monitoring SSLMate’s CRL-Watch website when we noticed that DigiCert had 4 CRLs listed (see below) with 404 error.

http://kr.crl.digicert.com/DigiCertSecureSiteKoreaEVCA.crl
http://kr.crl.digicert.com/DigiCertSecureSiteKoreaCA.crl
http://kr.crl.digicert.com/DigiCertSecureSiteKoreaECCCA.crl
http://kr.crl.digicert.com/DigiCertSecureSiteKoreaEVCA.crl

We were then notified by Apple that they were seeing some stale OCSP responses which lead to a larger investigation that resulted in finding that updates were not being pushed to our Content Delivery Network (CDN) in a timely manner.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times are MST
28th of Feb 2023 12:00 DigiCert migrates to new CDN for our CRL/OCSP
1st of March 2023 14:57 DigiCert personnel discovered a problem with 4 of our CRLs.
1st of March 2023 16:14 Our own internal alerting system that monitors SSLMate’s CRL-Watch website notified us of the problem.
1st of March 2023 16:16 Compliance confirmed the finding and escalated to Engineering.
1st of March 2023 17:37 Engineering resolved the misconfiguration and pushed out the change to our CDN.
1st of March 2023 17:51 Engineering started receiving the correct response from the affected domain.
2nd of March 2023 18:51 Apple notified DigiCert that they were seeing stale OCSP responses
2nd of March 2023 19:22 Investigation started into issue
2nd of March 2023 22:50 Manual push of OCSP occurs; ongoing Daily Manual pushes of OCSP to keep data fresh
5th of March 2023 20:08 Problem is isolated as an incorrect config or the origin point and resolution team is monitoring for further issues.
7th of March 2023 22:19 Issue is confirmed fixed

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The problems resulting from CRL misconfiguration and stale OCSP responses had no impact on certificates issued.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The problems resulting from CRL misconfiguration and stale OCSP responses had no impact on certificates issued.

5. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

On the 28th of February DigiCert changed CDN’s. During this, a number of configurations were missed. The kr.crl.digicert.com site was not migrated correctly so when the old CDN was shut down this started returning a 404 whilst we do monitor our CRL’s currently this is only a sample of all end points, and the KR ones were not in this.

Additionally the OCSP origin was for an older endpoint that was no longer in use. Whilst appearing valid, it was not updating every 24 hours as expected and then the cached response expired; currently we just pull a small sample of OCSP end points for expiration only and these had not yet expired.

6. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We will effectively manage changes and detect potentially any misconfigurations by adding change management auditing to all our CDN configurations. To do this going forward we will have a two party review process so an independent person will review the changes prior to deploying them. This has already been rolled out.

As additional improvements to our monitoring, we will add monitoring to our complete list of CRL distribution endpoints disclosed in CCADB rather than a sample set. Considering this is a large amount of CRL’s to add this will take some time.

Currently we have a validity of 7 days for our OCSP responses, however they are updated roughly every 24 hours. The changes in monitoring will be to make the following changes.

• When checking an OCSP response, we will validate age does not exceed 30 hours in addition to the expiration.
• Sample set should be sufficiently large and include critical certificates such as ICAs.
• In addition to logging/alerting on this information, it should also be visible via dashboard for our NOC
• Operations will have a standard playbook for responding to alerts of stale OCSP responses

We expect both of these to be done at the latest by end of Q2

Comment 4

[Martin Sullivan]

If there are no current questions, we will provide our next update on April 28th. DigiCert will be constantly monitoring in case there are any questions.

Comment 5

[Martin Sullivan]

Our CRL monitor has now been implemented.

This is monitoring all CRL's as per our list in CCADB for avaliblity and age to confrim we have the current one at all time.

The OCSP monitoring enhancements are still in progress and we plan to update this bug again at the end of the month.

Comment 6

[Martin Sullivan]

The OCSP monitoring was completed this week and is now live.

Re the CRL testing we had alerts on a soon to be expiring CRL’s pop in this week (as expected) so this is working in a real world environment

If there are any questions please let us know.

Comment 7

[Martin Sullivan]

With remediation now complete.
And it seems no further questions, Ben can we close this?

Comment 8

[Ben Wilson]

I intend to close this on 14-June-2023.