CommScope: Certificates were issued in which third-party web-based tools were used during validation.
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: nicol.so, Assigned: nicol.so)
Details
(Whiteboard: [ca-compliance] [dv-misissuance])
Attachments
(4 files)
This is a preliminary incident report. The full report will be published on or before 2024-06-21.
Incident Report (Preliminary)
Summary
CommScope: Certificates were issued in which third-party web-based tools were used during validation.
Impact
Certificates were issued not in compliance with the requirements of the CA/B Forum TLS Baseline Requirements. Seven certificates were revoked.
Timeline
All times are UTC-7.
2024-06-07:
- 10:58: CommScope’s external auditor verbally communicated to us that we had (valid) certificates issued in which the validation performed involved web-based tools operated by third parties and recommended CommScope reissue these certificates.
- 11:55: Initial investigation was started to assess the extent of the impact.
- 14:39: A message was sent to the incident report mailbox to trigger the incident response process.
- 15:00: Meeting was held to discuss the initial findings. A plan of action was agreed on. It was determined that the affected certificates must be revoked within 24 hours of the initial report.
- 15:28: A ticket was created in the internal issue tracking system.
- 16:44: Revocation of the affected certificates was approved by a CA officer.
- 17:58: Notification sent to subscribers to whom the affected certificates were issued.
- 18:12: Message sent to incident reporter including our findings and that the affected certificates would be revoked.
- 18:29: The affected certificates were revoked in CommScope’s CA system.
- 21:51: Revocation of the affected certificates were confirmed.
- 20:12: Confirmation of revocations sent to subscribers.
Root Cause Analysis
(to be added in the full report)
Lessons Learned
(to be added in the full report)
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Publish full report | Disclosure | 2024-06-21 |
Appendix
Details of affected certificates
| Issuer common name | Certificate serial number | SHA-256 checksum |
|---|---|---|
| CommScope Public Trust RSA Sub-CA-01-01 | 2B2AACD7D3C0FD607F801EB52780EA45C362B222 | d00f11a60518830083fdf0f9749da47146e6bc9cf9da61b160abecda17cdf989 |
| CommScope Public Trust RSA Sub-CA-02-01 | 4A5B9F6F3451374251D3AA428A7A5A5457C9BAFC | 7104c515b64909fb822d7c16ad668fba3cfdb693b4a60bad137912f5f12f06e6 |
| CommScope Public Trust ECC Sub-CA-01-01 | 642CF1D7329603AD2185C02830D54DCAF625BDA5 | 9598cb04ded047597e01ca0f42fc99b2b98a364075b65fc659ba35f03887dbd0 |
| CommScope Public Trust ECC Sub-CA-02-01 | 7EA4C215B5A129CACAB1C14C86EA653F5C5CFA4B | 0f640db65453862fbd923a8d8a4909b5327172ff18a1477535627646728e581b |
| CommScope Public Trust RSA Sub-CA-01-01 | 0E0C22D7AC3439CA6FD7679984FDAEC3068D5B36 | e25a16cc71549f28f5ba296913d39c076a191c3ca9b948247574c278770c304c |
| CommScope Public Trust RSA Sub-CA-01-01 | 6D2103448986FEFC4B2B8440F8E751096E39CDB0 | c8f824df84a28546c5cbb4f8c06934056a452321823676a4a5e1431de9a63acf |
| CommScope Public Trust RSA Sub-CA-01-01 | 7A7B3A76F65C897C9A6110C74EFF6655CAB605FD | 81b26f99eac6ac482d693d41e0d236140b87cbfba3988c93a800bfdffe2ac16b |
|
||
Comment 2•25 days ago
|
||
Hi Nicol -- Please ensure that the final incident report's timeline includes events leading up to the incident response, such as when these certificates were issued, when their information was validated, and when those validation processes were put in place.
|
||
Updated•25 days ago
|
|
||
Comment 3•24 days ago
|
||
(In reply to Nicol So from comment #1)
The three links on the bottom of this list are giving "Certificate not found" errors in crt.sh.
(In reply to Tim Callan from comment #3)
The three links on the bottom of this list are giving "Certificate not found" errors in crt.sh.
Those certificates are not logged in CT logs and for that reason not indexed by crt.sh. They are now included as attachments to this case.
|
||
Comment 8•24 days ago
|
||
I've just submitted those 3 certificates to Sectigo's Dodo log. Those crt.sh links now work.
(In reply to Nicol So from comment #7)
(In reply to Tim Callan from comment #3)
The three links on the bottom of this list are giving "Certificate not found" errors in crt.sh.
Those certificates are not logged in CT logs and for that reason not indexed by crt.sh. They are now included as attachments to this case.
How many more certificates are not logged in CT? I'm aware you're only in the Mozilla Root Store currently, but your own certificate policy does incorporate Apple, Chrome, and Microsoft by reference. More to the point in your Certificate Policy:
4.3.1.1.3 Generate and Submit Precertificates
Prior to issuing a Publicly-Trusted Certificate, a CA shall first generate a Precertificate, submit it to two different public Certificate Transparencies and add signed and timestamped responses from both Certificate Transparencies into a certificate extension of the TBSCertificate as defined in [18].
Even for attachment #9406502 I'm seeing a Domain-Validated cert (OID: 2.23.140.1.2.1) with Server Authentication (1.3.6.1.5.5.7.3.1). The certificate transparency timestamps for the last 3 only exist as they were submitted to Sectigo's Dodo log...
|
||
Updated•24 days ago
|
|
Assignee | |
Comment 10•18 days ago
|
||
(In reply to Aaron Gable from comment #2)
Hi Nicol -- Please ensure that the final incident report's timeline includes events leading up to the incident response, such as when these certificates were issued, when their information was validated, and when those validation processes were put in place.
Thanks, Aaron. We will include those details in our final incident report.
|
|
Assignee | |
Comment 11•17 days ago
|
||
(In reply to Wayne from comment #9)
(In reply to Nicol So from comment #7)
Those certificates are not logged in CT logs and for that reason not indexed by crt.sh. They are now included as attachments to this case.
How many more certificates are not logged in CT?
Prior to our public CAs being included in Mozilla's root program, we were unable to get CT log operators to include our precertificates in their production CT logs and as a result our public CAs issued 55 certificates that were not logged in a CT log. CommScope's public root CA certificates first appeared in Mozilla's NSS 3.95, released on 2023-11-16. Certificates issued by our public CAs after that date were logged in CT logs.
|
||
Comment 12•17 days ago
|
||
Understood. I haven't checked the issuance dates of the certificates in question, but do they predate this language existing on your CPS?
|
|
Assignee | |
Comment 13•14 days ago
|
||
Incident Report
Summary
Certificates were issued in which third-party web-based tools were used during validation. After investigation, a total of 84 certificates were identified as having been issued based on non-compliant validation, 7 of which were valid at the time CommScope was alerted to the issue. The rest were either expired or revoked at that time. The certificates were issued between 2021-06-04 and 2024-02-27.
Impact
Certificates were issued not in compliance with the requirements of the CA/B Forum TLS Baseline Requirements. Seven certificates, issued between 2023-09-08 and 2024-02-27, were revoked during incident response.
Timeline
All times are UTC-7.
2021-06-02:
- 17:14: Initial release of domain validation procedures, where the use of third-party web-based tools was part of the validation procedures.
2022-08-10:
- 15:26: Domain validation procedures last revised prior to the issuance of the certificates revoked in this incident. This was the version in effect when the validations for those certificates were performed. The use of third-party web-based tools was not changed in this revision.
2023-07-25:
- 13:15: Domain validation relied on by 4 of the certificates revoked as a result of the incident was performed.
2023-08-09:
- 14:30: Domain validation relied on by 3 of the certificates revoked as a result of the incident was performed.
2024-04-24:
- 15:53: Updated domain validation procedures published. The new procedures do not use non-authoritative servers operated by third parties.
2024-06-07:
- 11:25 (approx.): CommScope's external auditor verbally communicated to us that we had (valid) certificates issued in which the validation performed involved web-based tools operated by third parties and recommended that CommScope reissue these certificates.
- 11:55: Initial investigation was started to assess the extent of the impact.
- 14:39: A message was sent to the incident report mailbox to trigger the incident response process.
- 15:00: Meeting was held to discuss the initial findings. It was determined that the affected certificates must be revoked within 24 hours of the initial report. A plan of action was agreed on.
- 15:28: A ticket was created in the internal issue tracking system.
- 16:44: Revocation of the affected certificates was approved by a CA officer.
- 17:58: Notification sent to subscribers to whom the affected certificates were issued.
- 18:12: Message sent to incident reporter including our findings and that the affected certificates would be revoked.
- 18:29: The affected certificates were revoked in CommScope's CA system.
- 21:51: Revocation of the affected certificates were confirmed.
- 20:12: Confirmations of revocation sent to subscribers.
Root Cause Analysis
When our validation procedures were initially put in place in July 2021, we were not aware that web-based tools operated by third parties would be considered Delegated Third Parties. When our validation procedures were revised to not use non-authoritative servers operated by third parties, the action items generated did not include revisiting certificates issued before the change. This is a process issue, where the consideration of existing certificates and validations is not automatically part of the agenda when changes to the system are planned and implemented in response to incidents, changed requirements, or changed understanding of requirements.
Lessons Learned
When planning and implementing changes to a CA system, how the changes (and the underlying drivers) affect existing certificates and validations must be taken into consideration.
What went well
We revoked the affected certificates that were still valid within 24 hours of becoming aware of the issue.
What didn't go well
Existing certificates were not considered when the previous validation procedures were revised to be compliant.
Where we got lucky
The certificate revocations did not cause serious disruption for the subscribers.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Publish full incident report | Disclosure | 2024-06-21 |
| Revise process documentation so that whenever domain validation procedures are revised, existing validations and certificates are reviewed to determined whether they are affected by the underlying drivers of the revision. | Prevent | 2024-07-15 |
Appendix
Details of affected certificates
Certificates revoked during this incident:
| Issuer common name | Certificate serial number | SHA-256 checksum |
|---|---|---|
| CommScope Public Trust RSA Sub-CA-01-01 | 2B2AACD7D3C0FD607F801EB52780EA45C362B222 | d00f11a60518830083fdf0f9749da47146e6bc9cf9da61b160abecda17cdf989 |
| CommScope Public Trust RSA Sub-CA-02-01 | 4A5B9F6F3451374251D3AA428A7A5A5457C9BAFC | 7104c515b64909fb822d7c16ad668fba3cfdb693b4a60bad137912f5f12f06e6 |
| CommScope Public Trust ECC Sub-CA-01-01 | 642CF1D7329603AD2185C02830D54DCAF625BDA5 | 9598cb04ded047597e01ca0f42fc99b2b98a364075b65fc659ba35f03887dbd0 |
| CommScope Public Trust ECC Sub-CA-02-01 | 7EA4C215B5A129CACAB1C14C86EA653F5C5CFA4B | 0f640db65453862fbd923a8d8a4909b5327172ff18a1477535627646728e581b |
| CommScope Public Trust RSA Sub-CA-01-01 | 0E0C22D7AC3439CA6FD7679984FDAEC3068D5B36 | e25a16cc71549f28f5ba296913d39c076a191c3ca9b948247574c278770c304c |
| CommScope Public Trust RSA Sub-CA-01-01 | 6D2103448986FEFC4B2B8440F8E751096E39CDB0 | c8f824df84a28546c5cbb4f8c06934056a452321823676a4a5e1431de9a63acf |
| CommScope Public Trust RSA Sub-CA-01-01 | 7A7B3A76F65C897C9A6110C74EFF6655CAB605FD | 81b26f99eac6ac482d693d41e0d236140b87cbfba3988c93a800bfdffe2ac16b |
An additional 77 revoked/expired certificates, many of which not indexed by crt.sh, are provided in an attachment to this case.
|
|
Assignee | |
Comment 14•14 days ago
|
||
|
|
||
Comment 15•14 days ago
|
||
I'm still quite unclear on precisely what third-party tools were used and how they were used in the validation process. There was a similar incident in #1839305 less than a year from when this incident was confirmed. Consider that a useful reference in explaining what actually happened.
I am also interested in how that other incident did not prompt investigations for similar non-compliance. Or is this what prompted the revised domain validation procedures on 2024-04-24?
|
|
Assignee | |
Comment 16•10 days ago
|
||
(In reply to Wayne from comment #12)
Understood. I haven't checked the issuance dates of the certificates in question, but do they predate this language existing on your CPS?
The certificates with the CT log issue were issued when the language you quoted in comment #10 was in our CP/CPS.
|
|
Assignee | |
Comment 17•6 days ago
|
||
(In reply to Wayne from comment #15)
I'm still quite unclear on precisely what third-party tools were used and how they were used in the validation process. There was a similar incident in #1839305 less than a year from when this incident was confirmed. Consider that a useful reference in explaining what actually happened.
I am also interested in how that other incident did not prompt investigations for similar non-compliance. Or is this what prompted the revised domain validation procedures on 2024-04-24?
The third-party tools that were previous used in manual domain control validation include https://toolbox.googleapps.com/apps/dig/ and https://whois.domaintools.com/. The former was used in a manual procedure based on “DNS change”, to verify that the subscriber has created a prescribed TXT record (for the requested domain) containing a CommScope-supplied random value. The latter was used in an alternative manual procedure, to determine a domain contact email address for a requested domain via WHOIS lookup.
After further review of the information we have, we have identified additional events relevant to this case.
- 2023-12-21: Our external auditor noted to us that third-party web-based tools were considered delegated third parties. We started looking into replacing third-party web-based tools with locally-installed one. We did not have an appreciation of the implications on existing certificates and validations.
- 2024-01-18: Prompted by discussions preceding CA/B Forum SCWG Ballot SC-070, we started investigating how we would comply with requirements in the expected ballot. This redefined the search target for a replacement of third-party web-based tools. We anticipated updating our domain validation procedures when a solution was settled on.
- 2024-04-01: Work to revise domain validation procedures started.
|
|
||
Comment 18•6 days ago
|
||
(In reply to Nicol So from comment #17)
- 2023-12-21: Our external auditor noted to us that third-party web-based tools were considered delegated third parties. We started looking into replacing third-party web-based tools with locally-installed one. We did not have an appreciation of the implications on existing certificates and validations.
Okay your external auditor was actually paying attention to bugzilla then as that is when #1839305 explicitly mentioned using dig via the googleapp's toolbox:
We acknowledge that using an externally operated DNS tool like https://toolbox.googleapps.com/apps/dig/ for manual domain validation must be considered a Delegated Third Party (DTP). This is not clearly defined in the CA/Browser Forum Baseline Requirements (BR), but is a common understanding in the community.
Unfortunately this is nowhere near the first time this has been mentioned. As a quick overview:
2020-07-31 Izenpe: certificate issued to internal domain
2021-02-08 Microsoft PKI Services: Certificate Mis-Issuance, DNSNames must have a valid TLD
2023-06-15 Buypass: Domain validation using not allowed domain contact
Are you solely reliant on the third-party auditor to pass along potential BR problems? Is there any mention of these issues being checked internally by CommScope?
|
|
Assignee | |
Comment 19•3 days ago
|
||
(In reply to Wayne from comment #18)
Are you solely reliant on the third-party auditor to pass along potential BR problems? Is there any mention of these issues being checked internally by CommScope?
We do monitor for (upcoming) changes in the TLS BR and the emails on several mailing lists (CABF mailing lists, Mozilla dev-security-policy@mozilla.org, and CCADB Public). In this incident, we mentioned our external auditor because that happened to be the first channel through which we were made aware of the issue. We didn't take into account existing certificates and validations when we planned systems changes in response to the issue. We've learned our lesson and will be sure to consider existing certificates and validations when changes to the system are necessitated.
Description
•