2009491 - DigiCert: Several non-functioning AIA URLs

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[DigiCert]

Attachment: issuing_ca_ccadb.csv

Full Incident Report

Summary

• CA Owner CCADB unique ID: DigiCert

• Incident description: DigiCert received an external report that the URLs in two caIssuers AIA URI included in webPKI TLS certificates were non-functioning:
  http://cacerts.digicert.cn/DigiCertBasicOVG5TLSCNRSA4096SHA3842022CA1.crt
  http://cacerts.digicert.com/DigiCertQuoVadisTLSICAQVRootCA3G3.crt

• Timeline summary:

  • Non-compliance start date: The earliest notBefore of the affected CAs is 2021-01-06.

  • Non-compliance identified date: January 7, 2026

  • Non-compliance end date: January 8, 2026

• Relevant policies: Use of the authorityInformationAccess (AIA) extension for CAs is a SHOULD in section 7.1.2.5.1 of the TLS BR, with additional detail on formatting in 7.1.2.10.3. Under browser policies, CA operators are explicitly forbidden from issuing certificates containing cRLDistributionPoints or ocsp AIA extensions for which no operational CRL or OCSP service exists. DigiCert believes availability of the caIssuersAIA should also be considered required, even though the standards are silent on the matter.

• Source of incident disclosure: External report

Impact

• Total number of certificates: NA. 30 non-functioning caIssuers AIA downloads were identified.

• Total number of "remaining valid" certificates: NA.

• Affected certificate types: TLS and S/MIME.

• Incident heuristic: Visits to the URI in the caIssuers AIA resulted in a 404 error and did not download the CA certificate.

• Was issuance stopped in response to this incident, and why or why not?: No. The issue was rapidly remediated.

• Analysis: NA

• Additional considerations: NA

Timeline

Event	Date
External report received reporting two nonfunctioning caIssuers AIA	20260107 at 15:34 UTC
Issue confirmed	20260107 at 15:42 UTC
.crt downloads republished	20260107 at 17:20 UTC
Wider review completed; additional non-functioning caIssuers AIA identified and remediated	20260107 at 20:38 UTC
Verification that affected AIA republished	20260108 at 17:00 UTC

A complete list of the 30 affected public-trust caIssuers AIA is attached. Of those, 3 are TLS and 20 are S/MIME. The remaining 7 are Authentication or Timestamp.

Related Incidents

This Bug is similar to other recent bugs relating to accuracy of CRLDP filed in CCADB (see listing here).

Root Cause Analysis

Problem Statement

AIA caIssuers URLs returned 404 errors because .crt files for some CAs were not uploaded to the AIA repository server.

Why 1: Why did the AIA URLs return 404 errors?

Some CA certificate .crt files were not uploaded to the server hosting the AIA repository.

Why 2: Why were the CA certificate files not uploaded to the AIA repository server?

A manual step to upload the certificates to the AIA server was missed after CA creation.

Why 3: Why was the manual upload step missed?

There was no automated process, and steps on the manual checklist were missed that would have ensured the upload step was completed as part of the CA creation workflow. The process relied on human diligence to execute all required steps.

Why 4: Why was there no automated process or mandatory checklist for CA creation?

The CA creation workflow treats uploading the certificate for the caIssuers AIA URL as a separate, manual task rather than an automated step in the CA provisioning process.

Why 5: Why were process controls and automation not built into the CA creation workflow?

The CA creation process evolved organically. Although significant automation is deployed in linting, provisioning, audit record keeping, and CCADB reporting, the provisioning of the caIssuers AIA is a manual item as it requires separate publishing steps to the web/CDN as a security feature.

Root Cause #1: Missed step in manual process for two CAs.

Root Cause #2: Process design did not include upload of CA certificates to AIA repositories.

Root Cause #3: Process design did not include automated validation of caIssuers AIA URL accessibility.

Lessons Learned

• What went well: We reacted quickly to the external problem report, conducting a wider sweep than the reported issue.

• What didn’t go well: caIssuers AIA URI included in certificates for certain CAs did not download the CA certificate. Following the recent CRLDP bug, we had scheduled an investigation of all URI in CCADB; this problem report overtook us.

• Where we got lucky: The TLS CAs affected were very low volume CAs.

• Additional: Automation of CCADB interactions, as well as ongoing automated checks of data in CCADB filings, is desirable.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Update manual checklist with confirming signoff on AIA upload/functioning	Prevent	Root Cause # 1	Confirmation of completeness	2026-01-08	Complete
Confirm verification that all caIssuer AIA URLs resolve successfully (HTTP 200) and return valid certificates.	Prevent	Root Cause # 1	Confirmation of download	2026-01-08	Complete
Remediate identified caIssuerAIA download issues	Prevent	Root Cause # 1	Confirmation of download	2026-01-08	Complete
Automate the upload of CA certificates to AIA repositories	Prevent	Root Cause # 2	Confirmation of download	2026-02-01	Ongoing
Automate periodic check of functioning caIssuer AIA URI included in leaf certificates	Detect	Root Cause # 3	Confirmation of download	2026-02-01	Ongoing
Automate periodic check of URI included CCADB	Detect	Root Cause # 3	Confirmation of download	2026-01-23	Ongoing

Appendix

Relevant policy: TLS BR section 7.1.2.5.1 and section 7.1.2.10.3.

Comment 1

[chrome-root-program]

[In response to Comment 0]

1. It’s possible the evaluation and corrective actions were applied too narrowly. http://crl.digicert-validation.com/DigicertGlobalRootCA.crl, appearing in this certificate, is also 404’ing.

$ curl -f -s -S -O http://crl.digicert-validation.com/DigicertGlobalRootCA.crl
curl: (22) The requested URL returned error: 404

Was the scope of DigiCert’s investigation limited to AIA URLs? We do not observe discussion in this report related to CRLDPs except in relation to other bugs.

2. Related to the above, is DigiCert using its own database(s) or the CCADB as the canonical source of truth for URLs? The CRLDP listed in (1) is not disclosed to the CCADB, we assume due to a current limitation in the CCADB only being able to accept a single Full CRL for a CA record. Updates are being planned to allow the CCADB to collect multiple Full CRLs.

3. Some CCADB IRG formatting-related suggestions:

• The “CA Owner CCADB unique ID” intends to be the CCADB unique ID value begins with “A” followed by a six-digit number.
• The “Relevant policies” intends to be policy name(s), applicable version(s), and corresponding section(s) that result in this problem being diagnosed as an incident.
• The “Source of incident disclosure” intends to be a choice of “Self Reported”, “Third Party Reported”, or “Audit”.
• The “Impact” Section describes affected certificate types as TLS and S/MIME and the “Timeline” section states: “A complete list of the 30 affected public-trust caIssuers AIA is attached. Of those, 3 are TLS and 20 are S/MIME. The remaining 7 are Authentication or Timestamp.” Was there no impact to “Authentication or Timestamp”?

4. The “Relevant policies” Section states: “DigiCert believes availability of the caIssuersAIA should also be considered required, even though the standards are silent on the matter.”

Is DigiCert intending to raise this concern within the CA/Browser Forum Server Certificate Working Group or the Validation Subcommittee to improve the corresponding policy?

Or, is DigiCert planning on addressing this within its own PKI policy documents to hold itself to the standard it believes appropriate for the community?

5. The “Lessons Learned” Section states: “caIssuers AIA URI included in certificates for certain CAs did not download the CA certificate. Following the recent CRLDP bug, we had scheduled an investigation of all URI in CCADB; this problem report overtook us.”

The CCADB IRGs state (emphasis ours): “A list of things that caused the incident to have more impact than it otherwise would have, such as missing checks or unclear documentation. Each item here MUST have at least one corresponding Action Item below and SHOULD provide opportunities for others to ensure they make similar improvements if they haven’t already.”

The root cause for the investigation having been overtaken is unclear, as is any action associated with addressing this issue.

Comment 2

[DigiCert]

In Response to Comment 1:

1. It’s possible the evaluation and corrective actions were applied too narrowly. http://crl.digicert-validation.com/DigicertGlobalRootCA.crl, appearing in this certificate, is also 404’ing.

The indicated certificate for the CA named DigiCert Trust Service CA includes the URI http://crl.digicert-validation.com/DigiCertGlobalRootCA.crl which functions as intended. The curl provided in the comment uses a different URI.

The URI in the curl provided, however, was related to a CA which was cessationOfOperation revoked in CCADB in 2020. That CA, named DigiCert Trust Service ECC CA G1, never issued leaf certificates.

As noted previously, DigiCert has investigated all URLs included in currently valid certificates reported to CCADB and found no other issues.

2. Related to the above, is DigiCert using its own database(s) or the CCADB as the canonical source of truth for URLs? The CRLDP listed in (1) is not disclosed to the CCADB, we assume due to a current limitation in the CCADB only being able to accept a single Full CRL for a CA record. Updates are being planned to allow the CCADB to collect multiple Full CRLs.

DigiCert uses the CCADB API to synchronize our CCADB filings with our internal CA database (the break in synchronization arising from API changes being the cause of this incident). For our scanning, depending on the need, we are either comparing our internal database with CCADB, or comparing URI collected from issued certificates to CCADB.

We support the planned updates to allow the CCADB to optionally collect multiple Full CRLs.

3. Some CCADB IRG formatting-related suggestions:
   The “CA Owner CCADB unique ID” intends to be the CCADB unique ID value begins with “A” followed by a six-digit number.
   The “Relevant policies” intends to be policy name(s), applicable version(s), and corresponding section(s) that result in this problem being diagnosed as an incident.
   The “Source of incident disclosure” intends to be a choice of “Self Reported”, “Third Party Reported”, or “Audit”.
   The “Impact” Section describes affected certificate types as TLS and S/MIME and the “Timeline” section states: “A complete list of the 30 affected public-trust caIssuers AIA is attached. Of those, 3 are TLS and 20 are S/MIME. The remaining 7 are Authentication or Timestamp.” Was there no impact to “Authentication or Timestamp”?

Thank you for the feedback on IRG formatting; we’ll adopt this in future reports.

The requirements for Authentication and Timestamp CAs are not directly addressed in CABF standards (except to differentiate them from TLS CAs). There may have been an impact to users depending on the configuration of the relevant server and/or client, and client support for OCSP. However, we do not believe there was broad impact as these particular CAs serve specific customer use cases, and we believe that issues would have been rapidly flagged in our customer support channels.

4. The “Relevant policies” Section states: “DigiCert believes availability of the caIssuersAIA should also be considered required, even though the standards are silent on the matter.”
   Is DigiCert intending to raise this concern within the CA/Browser Forum Server Certificate Working Group or the Validation Subcommittee to improve the corresponding policy? Or, is DigiCert planning on addressing this within its own PKI policy documents to hold itself to the standard it believes appropriate for the community?

DigiCert will raise a discussion in the CABF Server Certificate WG to improve the corresponding policy.

5. The “Lessons Learned” Section states: “caIssuers AIA URI included in certificates for certain CAs did not download the CA certificate. Following the recent CRLDP bug, we had scheduled an investigation of all URI in CCADB; this problem report overtook us.”
   The CCADB IRGs state (emphasis ours): “A list of things that caused the incident to have more impact than it otherwise would have, such as missing checks or unclear documentation. Each item here MUST have at least one corresponding Action Item below and SHOULD provide opportunities for others to ensure they make similar improvements if they haven’t already.”
   The root cause for the investigation having been overtaken is unclear, as is any action associated with addressing this issue.

As noted, the expanded review of CCADB was scheduled to be performed as part of our Bug 2007219 report on December 30. This item was planned as a best practice, going beyond the root issues of that Bug. The new problem report described in this Bugzilla was filed 5 business days later on January 7, before that work could be completed. That work has now been completed and found no further issues in currently valid CAs.

Comment 3

[chrome-root-program]

[In response to Comment 2.]

Thanks for the response, but we’re having a hard time following the first answer. Can you help clarify?

In Comment 2 DigiCert stated:

The indicated certificate for the CA named DigiCert Trust Service CA includes the URI http://crl.digicert-validation.com/DigiCertGlobalRootCA.crl which functions as intended.

The certificate we referenced in Comment 1 includes http://crl.digicert-validation.com/DigiCertGlobalRootCA.crl in the CRLDP extension.

At the time of posting Comment 1, this URL was returning a 404 error via curl, as well as across multiple browsers. Since DigiCert’s response in Comment 2, we have continued to observe this behavior while attempting to access the URL across multiple machines, browsers (e.g., 1, 2, 3), and networks. This archive was taken at 16 Jan 2026 13:57:19 UTC - and also indicates a 404.

The curl provided in the comment uses a different URI.

The curl provided was:

$ curl -f -s -S -O http://crl.digicert-validation.com/DigicertGlobalRootCA.crl
curl: (22) The requested URL returned error: 404

The URI from the referenced certificate and the curl command are the same.

The URI in the curl provided, however, was related to a CA which was cessationOfOperation revoked in CCADB in 2020. That CA, named DigiCert Trust Service ECC CA G1, never issued leaf certificates.

Again, the URI in the curl provided was http://crl.digicert-validation.com/DigicertGlobalRootCA.crl and was derived from the example certificate provided. It’s not clear what role DigiCert Trust Service ECC CA G1 has with the reference certificate and curl command provided.

Comment 4

[Wayne]

The confusion may be arising as 'http://crl.digicert-validation.com/DigiCertGlobalRootCA.crl' is used in the CRLDP extension across 4 of DigiCert's intermediaries:
DigiCert Trust Service CA G1
DigiCert Trust Service ECC CA G2
DigiCert Trust Service ECC CA
DigiCert Trust Service CA

The first two are revoked via CRLSet, OneCRL, OCSP, and the CRL itself. If DigiCert did not follow the certificate linked and only consulted an internal list they may have jumped to conclusions.

I can concur that I'm getting 404s across some endpoints and routes too, but not all. There is definitely some network issue for DigiCert to investigate.

Comment 5

[Rob Stradling]

(In reply to chrome-root-program from comment #3)

[In response to Comment 2.]

Thanks for the response, but we’re having a hard time following the first answer. Can you help clarify?

Hi chrome-root-program. It's easy to miss, and it's also rather confusing, but these two URLs are not the same (lower-case 'c' vs upper-case 'C'):

http://crl.digicert-validation.com/DigiCertGlobalRootCA.crl
http://crl.digicert-validation.com/DigicertGlobalRootCA.crl

It looks like you saw one but cURL'd the other.

Comment 6

[chrome-root-program]

[In response to Comment 5.]

Thank you for clarifying, Rob.

The lowercase 'c' version appears in the revoked DigiCert mentioned in Comment 2.

@DigiCert team - our apologies for the mistake. Clearing the needinfo flag.

Comment 7

[DigiCert]

In Response to Comment 3 and Comment 4, DigiCert concurs with the observation in Comment 5: the two URI are different. We appreciate your clarification Rob. The URI in the active CA is functioning as intended.

We are on track to meet the action items outlined in our Full Incident Report. We request the “Next update” Whiteboard field for this Bug to be set for 2026-02-02.

Comment 8

[DigiCert]

We are on track to meet the action items outlined in our Full Incident Report and expect to provide a final update next week.

Comment 9

[DigiCert]

We’ve completed all action items and plan to post a Closure Summary by early next week.

Comment 10

[DigiCert]

Report Closure Summary

Incident description

DigiCert was notified of two non-functioning caIssuers AIA URLs in webPKI TLS certificates that returned 404 errors. Upon investigation, a total of 30 non-functioning caIssuers AIA URLs were identified across various certificate types, including 3 TLS, 20 S/MIME, and 7 Authentication/Timestamp. The affected URLs prevented proper CA certificate downloads, with the issue dating back to the earliest certificate issuance on 2021-01-06.

Incident Root Cause(s)

The primary root cause was a missed manual step during CA creation where .crt files were not uploaded to the AIA repository server. Additionally, the CA creation workflow lacked automation for these uploads, and there was no automated validation to ensure caIssuers AIA URLs were accessible and returned HTTP 200 responses. This reliance on manual processes without sufficient controls led to the oversight.

Remediation description

All affected .crt files were republished to the AIA repositories, resolving the two initially reported URLs by 2026-01-07 at 17:20 UTC and the remaining 28 by 20:38 UTC the same day. A comprehensive review and verification confirmed functionality of all 30 URLs by 2026-01-08 at 17:00 UTC. No further issues were found in currently valid certificates.

Commitment summary

Ongoing automation of periodic checks for caIssuers AIA URL accessibility in leaf certificates to prevent future oversights.

Continuous automation of URL validations included in CCADB filings.

Advocacy in the CA/Browser Forum Server Certificate Working Group to propose explicit requirements for caIssuers AIA availability in policy standards.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 11

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-02-16.