1930759 - DigiCert: Domain used for CRLs and OCSP has expired

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

Several of DigiCert's certificates contain OCSP and CRL URLs under digicert-validation.com, e.g. https://crt.sh/?sha256=72D0F7CBF4529B80C34A7CBB438BD1D0E1FC26E80E59CBD4FD7314FCDDF0E994

The hostnames are not currently resolving. It appears that the domain expired yesterday:

Registrar Registration Expiration Date: 2024-11-11T05:18:33Z

Comment 1

[Tim Hollebeek]

SUMMARY

One of the domains that Digicert uses to host CRLs and an OCSP responder expired which caused a short outage of these services for the affected ICAs.

IMPACT

7 ICA’s were impacted, none of these have active end user certificates.
4 currently active
3 revoked

TIMELINE -
All times are in UTC

23:59 11 Nov 2024 Domain registration expired for digicert-validation.com
11:36 12 Nov 2024 DigiCert internal monitoring alerted that the CRL and OCSP are not responding
20:41 12 Nov 2024 DigiCert were informed that OCSP and CRL unavailability via this bug
22:15 12 Nov 2024 Domain was renewed and services were brought back up.

ROOT CAUSE ANALYSIS

The domain digicert-validation.com was one of 2 domains being managed by a separate service for the China market (along with digicert.cn). These domains need to be managed by a Chinese national and were for a product opportunity we ended up not pursuing. Because of the non-standard management of these domains, the renewal notices from this service were missed and the domain expired.

LESSONS LEARNED

WHAT WENT WELL

The domain was registered successfully.

WHAT DIDN'T GO WELL

Whilst the internal monitoring did pick up the problem, it went to a team that isn’t staffed 24/7.

WHERE WE GOT LUCKY

This impacted a small number of ICAs with no active leaf certificates.

ACTION ITEMS

Action Item	Kind	Due Date
Implement automated collection of centralized list of domains included in AIA/CDP fields in issued certificates	Prevent	2025-01-31
Make sure all such domains are periodically checked for status and alerts are forwarded to channels that are monitored 24/7	Prevent	2025-01-31

APPENDIX

DETAILS OF AFFECTED CERTIFICATES

DigiCert Trust Service CA G1 https://crt.sh/?sha256=64616B62820D458F7C71E3449ECCF3F3E4735B76879860C1C660DF23438607EA

DigiCert High Assurance Trust Service ECC EV CA https://crt.sh/?sha256=E9529B428FB67390BC6455D79BA2434E816C54FE4F359930CB709DB256FDDF94%09

DigiCert Trust Service ECC CA G1 https://crt.sh/?sha256=017CBB9C3BF9F115E6DB26E47C90EAFA6A7FAEE4F7D348AE2808E1C7A8CCB107

DigiCert High Assurance Trust Service EV CA https://crt.sh/?sha256=72D0F7CBF4529B80C34A7CBB438BD1D0E1FC26E80E59CBD4FD7314FCDDF0E994

DigiCert Trust Service CA https://crt.sh/?sha256=75AC8E41D9A7CC758D3998FE030F638CFD28855823DA4E9B56954CFBDE054EB6%09+

DigiCert Trust Service ECC CA https://crt.sh/?sha256=F269AC00B410003F72DC628AFB3D950279630C7C5D0C82148A0FD24DF4DA4301

DigiCert Trust Service ECC CA G2 https://crt.sh/?sha256=B5FECC2F696C850323FC7A85C65B50A4C5E1BFBF823E266AB0A46F013B2C0EB2

Comment 2

[Tim Hollebeek]

Watching for questions.

Comment 3

[Tim Hollebeek]

Unless the questions are urgent, they will be answered next week.

Comment 4

[Tim Hollebeek]

Still watching.

Comment 5

[Kiel C]

The timeline here begins at the point of domain expiry - but that seems to ignore the root cause, which certainly must've been some time earlier than expiry? For me to learn the most I can from this report, it would be helpful to have a complete timeline that includes some context for the root cause of this issue. What lead to the choice for this domain to be managed in non-standard way? Are there currently other situations which have lead to non-standard domain management?

Comment 6

[Tim Hollebeek]

The timeline here begins at the point of domain expiry - but that seems to ignore the root cause, which certainly must've been some time earlier than expiry? For me to learn the most I can from this report, it would be helpful to have a complete timeline that includes some context for the root cause of this issue. What lead to the choice for this domain to be managed in non-standard way? Are there currently other situations which have lead to non-standard domain management?

The original domains was registered by an agent in China as the domain was going to be used for a Chinese project. We have several domains that were registered through the same agent that we are trying to transfer to DigiCert. The issue is that Aliyun (where the domains were registered) seems hesitant to transfer the domains to a non-Chinese entity.

There wasn’t really a “choice” made to operate the domains in a non-standard way. The Chinese sales person registered the domains with Aliyun and then kicked off the project with the product team.

Comment 7

[Tim Hollebeek]

We will be updating bugs on Mondays over the holidays.

Comment 8

[Tim Hollebeek]

Happy holidays!

Comment 9

[Tim Hollebeek]

Onward to 2025!

Comment 10

[Tim Hollebeek]

Nothing new here.

Comment 11

[DigiCert]

Going forward, we will be posting updates as DigiCert, as suggested by a proposed update to CCADB policy. This will make it clear these are official DigiCert responses.

Comment 12

[DigiCert]

The action items are on track to be completed by 1/31.

Comment 13

[DigiCert]

Still on track.

Comment 14

[DigiCert]

The 1-31 action items are complete.

Comment 15

[DigiCert]

DigiCert completed all action items listed for this bug.

Action Items

Action Item	Kind	Due Date
Implement automated collection of centralized list of domains included in AIA/CDP fields in issued certificates	Prevent	2025-01-31
Make sure all such domains are periodically checked for status and alerts are forwarded to channels that are monitored 24/7	Prevent	2025-01-31

Incident Report Closure Summary
Incident Description:
One of the domains that Digicert uses to host CRLs and an OCSP responder expired which caused a short outage of these services for the affected ICAs.

Incident Root Cause(s):
The domain digicert-validation.com was one of 2 domains being managed by a separate service for the China market (along with digicert.cn). These domains need to be managed by a Chinese national and were for a product opportunity we ended up not pursuing. Because of the non-standard management of these domains, the renewal notices from this service were missed and the domain expired.

Remediation Description:
We are now tracking in an automated way what domains are included in AIA/CDP fields for issued certificates and making sure all such domains are monitored. This will make sure that all such domains are monitored, even if they are no longer being actively used for product development.

Commitment Summary:
We will continue to use this new automated system to prevent similar issues from recurring in the future.

@Ben Wilson – As all action items are completed, will you please close this bug?

Comment 16

[Ben Wilson]

I'll schedule to close this on Wed. 12-Feb-2025.