DigiCert: Unclear Disclosure of CAA Issuer Domain Names
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: agwa-bugs, Assigned: tim.hollebeek)
Details
(Whiteboard: [ca-compliance] [policy-failure] [external])
Attachments
(1 file)
22.25 KB,
text/csv
|
Details |
DigiCert's CP/CPS lists the following as a recognized CAA Issuer Domain Name in Section 4.2:
or registered country jurisdictions (e.g., digicert.de).
It's not clear what this means.
Does DigiCert recognize any Issuer Domain Name of the form digicert.XX
?
Or do they have a list of domains which they own, and only recognize Issuer Domain Names on that list? If so, the CP/CPS doesn't satisfy BR 2.2's requirement to "clearly specify the set of Issuer Domain Names that the CA recognizes", as it's not generally possible for a third party to determine if a particular domain is registered to DigiCert or not. So someone reading this CP/CPS can't actually determine what Issuer Domain Names DigiCert accepts.
Updated•5 months ago
|
Assignee | ||
Comment 1•5 months ago
|
||
Thank you for the report, we are looking into it now.
Assignee | ||
Comment 2•5 months ago
|
||
Incident Report
Summary
This is a preliminary report, and we will provide a full incident report next week. While investigating this report, we discovered a related issue and are still analyzing that issue. The full incident report will address both the posting from Andrew Ayer and the additional item currently being researched.
An external researcher reported that our CPS contained language suggesting that we sometimes use domains of the form “digicert.XX” as CAA domains, where XX is a country code. The reality is that DigiCert’s CAA checking code does not permit issuance using any such domains as CAA domains. This language was added during the early days of CAA and kept in case subscribers wanted to use a local version of the CAA record. As the idea was never implemented in code, we removed the language from the CPS.
While investigating the issue and comparing our implementation, CPS, and CCADB disclosures, we noticed that ‘symantec.com’ had been inadvertently removed from our CPS as a CAA domain during a recent update. We’ve already updated the CPS to re-include the deleted CAA information as the removal was unintentional and are looking into both the root cause and any impacted certificates. More details will be provided in a subsequent update.
Impact
The unclear CPS language described behavior that we never implemented on DigiCert’s systems. As there are no certificates relying on the unclear CAA specification, no revocation is required.
We are still investigating the impact of omitting ‘symantec.com’ during the last CPS update.
Assignee | ||
Comment 3•5 months ago
|
||
As noted above, 'symantec.com' was missing from our CPS from July 29 to August 27. For certificates that relied on a CAA record from 'symantec.com' during that time period, the certificates were not issued in full compliance with our CPS, and need to be revoked in accordance with TLS BR 4.9.1.1 item 12, which is a 5-day revocation. Affected serial list attached, and revocations initiated at 19:29 UTC today.
Assignee | ||
Comment 4•5 months ago
|
||
Affected serial list using 'symantec.com' as a CAA domain.
Updated•5 months ago
|
Assignee | ||
Comment 5•5 months ago
|
||
All certificates were revoked by 3rd sept 19:05 UTC.
Assignee | ||
Comment 6•5 months ago
|
||
Full report is being finalized will be posted next week.
Assignee | ||
Comment 7•5 months ago
|
||
Incident Report
Summary
For a brief period of time, DigiCert’s public CPS and implementation differed as to which CAA domains are used to approve issuance by DigiCert. The CPS referenced the use of “digicert.XX” domains with different country codes. Whether such generalized descriptions are allowed or not is debatable, but also irrelevant as using such domains is not an existing DigiCert practice. Therefore, we removed the unnecessary text from the CPS.
As is standard procedure when we receive externally reported issues, we look around in the area for additional areas for improvement. During such a review, it was noticed that the CAA domain ‘symantec.com’ was inadvertently removed during a recent and large CPS re-organization. The error has been reversed, however 185 certificates were issued relying on CAA records that specified ‘symantec.com’ during the time period when ‘symantec.com’ was not in the CPS. Due to the reliance on a CAA issuance domain that was not in the CPS at the time of issuance, we are declaring these certificates misissued and replacing them.
Impact
DigiCert revoked the misissued certificates. This is a five-day revocation that falls under reason 12 in TLS BR section 4.9.1.1, “The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded).” We researched previous similar but more severe CAA incidents at other CAs, and confirmed they reached the same determination.
Timeline
All times are UTC.
29 July 2024 00:00 Digicert CP/CPS v 7.0 comes into effect. This major revamp has mistakenly dropped the symantec.com domain from approved CAA domains.
26 August 2024 13:21 Andrew posts this bug kicking off an internal investigation
26 August 2024 18:28 Confirmed that no instances of digicert.xx were being used in our systems (excluding digicert.com which is listed separately)
26 August 2024 22:17 during the investigation it was noted there was a difference in the list of approved CAA domains in our CA system and the CPS in that the symantec.com domain was missing from the CPS this kicked off a secondary investigation
27 August 2024 19:23 Digicert CP/CPS 7.01 is uploaded adding back in the symantec.com CAA domain
27 August 2024 22:00 Meeting to confirm that if there was issuance using Symantec.com in the period of the 7.0 CP/CPS that this would be a misissuance and a scan was started to confirm if there were any instances of this.
29 August 2024 19:29 investigation completed and 185 certificates found, these are scheduled for revocation.
3 September 2024 19:05 Last Certificate revoked.
Root Cause Analysis
In order to make our CPS easier to review and maintain, the information is being transitioned from traditional documents to an automated system based on asciidoc, similar to how the BRs themselves are maintained. The CPS update went through all the correct procedures and reviews, but due to the fact that the document was transitioning from a traditional document to ASCIIDOC, there were a large number of changes due to that and, this editorial error went unnoticed. Additionally, review of the staff responsible for CPS approvals found that too few individuals with strong technical knowledge were involved in reviews.
Lessons Learned
What went well
-
We got an external bug report, realized it suggested that the area wasn’t sufficiently reviewed, and did a comprehensive review of the CPS, CAA implementation, documentation, and CCADB disclosures to find any additional problems.
-
We found and fixed an additional problem.
-
The incident process ran smoothly despite recent organizational, staffing and leadership changes.
-
All certificates revoked within the BR mandated period of time.
What didn't go well
- We didn’t find the problem during the reviews when it should have been found. In hindsight, despite the change being intended to have no substantive impact, more effort should have been put into this review due to its complicated nature.
Where we got lucky
- We try not to rely on getting lucky, because it happens unpredictably. We’d rather be good than lucky.
Action Items
| Action Item | Kind | Due Date |
| ----------- | ---- | -------- |
|Additional technical reviewers to CPS approval process | Detect | 2024-09-15|
|Add automated checks to make sure CPS CAA domains and CAA implementation are consistent at all times. | Prevent | 2024-10-31|
Appendix
Details of affected certificates
See caa-incident.csv
Assignee | ||
Comment 8•5 months ago
|
||
Just an update that the 2024-09-15 remediation was completed on time. I was added as a reviewer in our CPS review process, as were a few other technical experts from our engineering organization.
Assignee | ||
Comment 9•4 months ago
|
||
We've got a pretty good handle on the design for how we're going to build the automation to compare the CPS and CAA implementation. We'll have more information in mid-October when we're putting it into place. We're leveraging the fact that we've moved to an ASCIIDOC-based process for managing our CPS, so it is more amenable to automated comparisons than the traditional CPSs we were previously using.
Assignee | ||
Comment 10•4 months ago
|
||
Development work continues.
Updated•4 months ago
|
Assignee | ||
Comment 11•4 months ago
|
||
Development tasks are all in progress or in testing, and no risks to the deployment schedule have been identified. We expect the 10/31 milestone to be it.
Assignee | ||
Comment 12•4 months ago
|
||
Still looking good for 10/31.
Updated•3 months ago
|
Assignee | ||
Comment 13•3 months ago
|
||
An issue was found shortly before deployment, and is being addressed. Deployment is not anticipated to be complete by 2024-11-08.
Comment 14•3 months ago
|
||
Can we get some more insight into the issue found, in case there's a learning opportunity there?
(Also, I assume you meant to say "Deployment is now anticipated..."?)
Assignee | ||
Comment 15•3 months ago
|
||
Yep, "now".
Happy to provide more info on the issue, though it's not particularly insightful. I'm hearing there was an error in the deployment instructions, and the deployment had to be rolled back, re-reviewed, and scheduled for the next deployment window.
Assignee | ||
Comment 16•3 months ago
|
||
The fix is deployed now. There are now automated comparisons in place that compare our CPS with the configuration of the validation system.
Assignee | ||
Comment 17•3 months ago
|
||
We have nothing further, are we done here?
Comment 18•3 months ago
|
||
Hi Tim,
Can you provide a Closing Summary and request that this be closed?
Thanks!
Ben
A closing summary should briefly:
- describe the incident, its root cause(s), and remediation;
- summarize any ongoing commitments made in response to the incident; and
- attest that all Action Items have been completed.
Here is a template:
Incident Report Closure Summary
- Incident Description: [Two or three sentences summarizing the incident.]
- Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
- Remediation Description: [Two or three sentences summarizing the incident's remediation.]
- Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]
All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.
Comment 19•3 months ago
|
||
Re: Comment #18, I'll note that this a request and not a requirement yet.
Assignee | ||
Comment 20•2 months ago
|
||
Thank you, we have some comments and concerns about this requirement that we will submit as part of the policy review process.
But in the meantime, we're willing to give it a try on this bug and see how it goes ... for science. Hopefully it goes well.
Assignee | ||
Comment 21•2 months ago
|
||
Incident Description:
The original bug was opened due to some unused and outdated language in the CPS, which was simply removed. However, while looking into the issue, we discovered that symantec.com had been omitted from the CAA domains for approximately a month. The certificates issued in reliance on the missing CAA record were revoked and replaced.
Incident Root Cause(s):
The CPS underwent a fairly large revision while being transitioned to a more modern, GitHub/ASCII-based production system. Reviewing via diffs was not possible, as the CPS was transitioning from one format to another. The one line got lost and was not noticed subsequent reviews.
Remediation Description:
Part of the problem is that the existing reviewers tended to be policy folks, so more reviewers were added from engineering who can catch technical errors in CPS changes. We also built an automated system that is capable of comparing the CAA domain list in the CPS to the CAA domain list in our validation system, and will flag any discrepancies in the future.
All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.
Assignee | ||
Comment 22•2 months ago
|
||
Closing summary posted.
Assignee | ||
Comment 23•2 months ago
|
||
We have nothing further.
Assignee | ||
Comment 24•2 months ago
|
||
Happy holidays.
Assignee | ||
Comment 25•1 month ago
|
||
Onward to 2025!
Assignee | ||
Comment 26•1 month ago
|
||
Still requesting that this be closed. We have posted the closing summary as requested.
Comment 27•1 month ago
|
||
Closing this now, as no comments or questions have been presented.
Description
•