Closed Bug 1914911 Opened 5 months ago Closed 1 month ago

DigiCert: Unclear Disclosure of CAA Issuer Domain Names

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: agwa-bugs, Assigned: tim.hollebeek)

Details

(Whiteboard: [ca-compliance] [policy-failure] [external])

Attachments

(1 file)

DigiCert's CP/CPS lists the following as a recognized CAA Issuer Domain Name in Section 4.2:

or registered country jurisdictions (e.g., digicert.de).

It's not clear what this means.

Does DigiCert recognize any Issuer Domain Name of the form digicert.XX?

Or do they have a list of domains which they own, and only recognize Issuer Domain Names on that list? If so, the CP/CPS doesn't satisfy BR 2.2's requirement to "clearly specify the set of Issuer Domain Names that the CA recognizes", as it's not generally possible for a third party to determine if a particular domain is registered to DigiCert or not. So someone reading this CP/CPS can't actually determine what Issuer Domain Names DigiCert accepts.

Assignee: nobody → tim.hollebeek
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]

Thank you for the report, we are looking into it now.

Incident Report

Summary

This is a preliminary report, and we will provide a full incident report next week. While investigating this report, we discovered a related issue and are still analyzing that issue. The full incident report will address both the posting from Andrew Ayer and the additional item currently being researched.

An external researcher reported that our CPS contained language suggesting that we sometimes use domains of the form “digicert.XX” as CAA domains, where XX is a country code. The reality is that DigiCert’s CAA checking code does not permit issuance using any such domains as CAA domains. This language was added during the early days of CAA and kept in case subscribers wanted to use a local version of the CAA record. As the idea was never implemented in code, we removed the language from the CPS.

While investigating the issue and comparing our implementation, CPS, and CCADB disclosures, we noticed that ‘symantec.com’ had been inadvertently removed from our CPS as a CAA domain during a recent update. We’ve already updated the CPS to re-include the deleted CAA information as the removal was unintentional and are looking into both the root cause and any impacted certificates. More details will be provided in a subsequent update.

Impact

The unclear CPS language described behavior that we never implemented on DigiCert’s systems. As there are no certificates relying on the unclear CAA specification, no revocation is required.

We are still investigating the impact of omitting ‘symantec.com’ during the last CPS update.

As noted above, 'symantec.com' was missing from our CPS from July 29 to August 27. For certificates that relied on a CAA record from 'symantec.com' during that time period, the certificates were not issued in full compliance with our CPS, and need to be revoked in accordance with TLS BR 4.9.1.1 item 12, which is a 5-day revocation. Affected serial list attached, and revocations initiated at 19:29 UTC today.

Attached file caa-incident.csv

Affected serial list using 'symantec.com' as a CAA domain.

Whiteboard: [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure] [external]

All certificates were revoked by 3rd sept 19:05 UTC.

Full report is being finalized will be posted next week.

Incident Report

Summary

For a brief period of time, DigiCert’s public CPS and implementation differed as to which CAA domains are used to approve issuance by DigiCert. The CPS referenced the use of “digicert.XX” domains with different country codes. Whether such generalized descriptions are allowed or not is debatable, but also irrelevant as using such domains is not an existing DigiCert practice. Therefore, we removed the unnecessary text from the CPS.

As is standard procedure when we receive externally reported issues, we look around in the area for additional areas for improvement. During such a review, it was noticed that the CAA domain ‘symantec.com’ was inadvertently removed during a recent and large CPS re-organization. The error has been reversed, however 185 certificates were issued relying on CAA records that specified ‘symantec.com’ during the time period when ‘symantec.com’ was not in the CPS. Due to the reliance on a CAA issuance domain that was not in the CPS at the time of issuance, we are declaring these certificates misissued and replacing them.

Impact

DigiCert revoked the misissued certificates. This is a five-day revocation that falls under reason 12 in TLS BR section 4.9.1.1, “The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded).” We researched previous similar but more severe CAA incidents at other CAs, and confirmed they reached the same determination.

Timeline

All times are UTC.

29 July 2024 00:00 Digicert CP/CPS v 7.0 comes into effect. This major revamp has mistakenly dropped the symantec.com domain from approved CAA domains.

26 August 2024 13:21 Andrew posts this bug kicking off an internal investigation

26 August 2024 18:28 Confirmed that no instances of digicert.xx were being used in our systems (excluding digicert.com which is listed separately)

26 August 2024 22:17 during the investigation it was noted there was a difference in the list of approved CAA domains in our CA system and the CPS in that the symantec.com domain was missing from the CPS this kicked off a secondary investigation

27 August 2024 19:23 Digicert CP/CPS 7.01 is uploaded adding back in the symantec.com CAA domain

27 August 2024 22:00 Meeting to confirm that if there was issuance using Symantec.com in the period of the 7.0 CP/CPS that this would be a misissuance and a scan was started to confirm if there were any instances of this.

29 August 2024 19:29 investigation completed and 185 certificates found, these are scheduled for revocation.

3 September 2024 19:05 Last Certificate revoked.

Root Cause Analysis

In order to make our CPS easier to review and maintain, the information is being transitioned from traditional documents to an automated system based on asciidoc, similar to how the BRs themselves are maintained. The CPS update went through all the correct procedures and reviews, but due to the fact that the document was transitioning from a traditional document to ASCIIDOC, there were a large number of changes due to that and, this editorial error went unnoticed. Additionally, review of the staff responsible for CPS approvals found that too few individuals with strong technical knowledge were involved in reviews.

Lessons Learned

What went well

  • We got an external bug report, realized it suggested that the area wasn’t sufficiently reviewed, and did a comprehensive review of the CPS, CAA implementation, documentation, and CCADB disclosures to find any additional problems.

  • We found and fixed an additional problem.

  • The incident process ran smoothly despite recent organizational, staffing and leadership changes.

  • All certificates revoked within the BR mandated period of time.

What didn't go well

  • We didn’t find the problem during the reviews when it should have been found. In hindsight, despite the change being intended to have no substantive impact, more effort should have been put into this review due to its complicated nature.

Where we got lucky

  • We try not to rely on getting lucky, because it happens unpredictably. We’d rather be good than lucky.

Action Items

| Action Item | Kind | Due Date |

| ----------- | ---- | -------- |

|Additional technical reviewers to CPS approval process | Detect | 2024-09-15|

|Add automated checks to make sure CPS CAA domains and CAA implementation are consistent at all times. | Prevent | 2024-10-31|

Appendix

Details of affected certificates

See caa-incident.csv

Just an update that the 2024-09-15 remediation was completed on time. I was added as a reviewer in our CPS review process, as were a few other technical experts from our engineering organization.

We've got a pretty good handle on the design for how we're going to build the automation to compare the CPS and CAA implementation. We'll have more information in mid-October when we're putting it into place. We're leveraging the fact that we've moved to an ASCIIDOC-based process for managing our CPS, so it is more amenable to automated comparisons than the traditional CPSs we were previously using.

Development work continues.

Whiteboard: [ca-compliance] [policy-failure] [external] → [ca-compliance] [policy-failure] [external] Next update 2024-10-15

Development tasks are all in progress or in testing, and no risks to the deployment schedule have been identified. We expect the 10/31 milestone to be it.

Still looking good for 10/31.

Whiteboard: [ca-compliance] [policy-failure] [external] Next update 2024-10-15 → [ca-compliance] [policy-failure] [external] Next update 2024-11-01

An issue was found shortly before deployment, and is being addressed. Deployment is not anticipated to be complete by 2024-11-08.

Can we get some more insight into the issue found, in case there's a learning opportunity there?

(Also, I assume you meant to say "Deployment is now anticipated..."?)

Yep, "now".

Happy to provide more info on the issue, though it's not particularly insightful. I'm hearing there was an error in the deployment instructions, and the deployment had to be rolled back, re-reviewed, and scheduled for the next deployment window.

The fix is deployed now. There are now automated comparisons in place that compare our CPS with the configuration of the validation system.

We have nothing further, are we done here?

Hi Tim,
Can you provide a Closing Summary and request that this be closed?
Thanks!
Ben

A closing summary should briefly:

  • describe the incident, its root cause(s), and remediation;
  • summarize any ongoing commitments made in response to the incident; and
  • attest that all Action Items have been completed.

Here is a template:

Incident Report Closure Summary

  • Incident Description: [Two or three sentences summarizing the incident.]
  • Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
  • Remediation Description: [Two or three sentences summarizing the incident's remediation.]
  • Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Flags: needinfo?(tim.hollebeek)

Re: Comment #18, I'll note that this a request and not a requirement yet.

Whiteboard: [ca-compliance] [policy-failure] [external] Next update 2024-11-01 → [ca-compliance] [policy-failure] [external]

Thank you, we have some comments and concerns about this requirement that we will submit as part of the policy review process.

But in the meantime, we're willing to give it a try on this bug and see how it goes ... for science. Hopefully it goes well.

Incident Description:

The original bug was opened due to some unused and outdated language in the CPS, which was simply removed. However, while looking into the issue, we discovered that symantec.com had been omitted from the CAA domains for approximately a month. The certificates issued in reliance on the missing CAA record were revoked and replaced.

Incident Root Cause(s):

The CPS underwent a fairly large revision while being transitioned to a more modern, GitHub/ASCII-based production system. Reviewing via diffs was not possible, as the CPS was transitioning from one format to another. The one line got lost and was not noticed subsequent reviews.

Remediation Description:

Part of the problem is that the existing reviewers tended to be policy folks, so more reviewers were added from engineering who can catch technical errors in CPS changes. We also built an automated system that is capable of comparing the CAA domain list in the CPS to the CAA domain list in our validation system, and will flag any discrepancies in the future.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Flags: needinfo?(tim.hollebeek)

Closing summary posted.

We have nothing further.

Happy holidays.

Onward to 2025!

Still requesting that this be closed. We have posted the closing summary as requested.

Closing this now, as no comments or questions have been presented.

Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: