1952591 - CommScope: Validation issue in SCT extensions in certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Nicol So]

Preliminary Incident Report

Summary

• Incident description: CommScope discovered that the SCT extensions in certificates issued by its public CAs fail validation
• Relevant policies: CommScope's CP/CPS
• Source of incident disclosure: CommScope's internal investigation

Comment 1

[Ben Wilson]

What type of certificates were these--DV, OV, EV, or a combination?

Comment 2

[Nicol So]

(In reply to Ben Wilson from comment #1)

What type of certificates were these--DV, OV, EV, or a combination?

The affected certificates are all DV certificates. Note that despite the word "validation" in the summary, the issue is NOT related to the validation of domain control or subject information.

Comment 3

[Nicol So]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A010155
• Incident description: CommScope discovered that the SCTs (Signed Certificate Timestamps) in certificates issued by its public CAs failed signature verification, even though the certificates were logged succesfully to certificate transparency logs accepted by Chrome, Apple, and Mozilla root programs
• Timeline summary:
  • Non-compliance start date: 2021-06-04
  • Non-compliance identified date: 2025-03-04
  • Non-compliance end date: 2025-03-10
• Relevant policies: CommScope CP/CPS
• Source of incident disclosure: Self (CommScope discovered the issue itself)

Impact

• Total number of certificates: 49
• Total number of "remaining valid" certificates: 0
• Affected certificate types: DV (domain-validated)
• Incident heuristic: The full corpus of affected certificates is disclosed in the Appendix
• Was issuance stopped in response to this incident, and why or why not?: Yes, we made a decision that no more certificates from our public CAs will be issued until the issue has been corrected.

Timeline

All times UTC-0700 unless otherwise noted.

2021-04-26:

• Revision date of CommScope's combined CP/CPS revision 2.0, in which requirements about publication to CT logs first appeared.

2021-05-20:

• Testing of the first release of CA application with support for certificate transparency started.

2021-05-26:

• Deployment of the first release of CA application with support for certificate transparency. The code containing incorrect logic for the construction of SCTs was introduced in this release.

2021-06-04:

• 11:13 First non-compliant certificate with the problem was issued.

2025-03-04:

• 13:30 An internal code review, which included the code for SCT extension construction, revealed an error in the SCT extension construction logic.

2025-03-05:

• A fix for the incorrect logic was tested and verified.

2025-03-10:

• 15:22 (Approx.) Seven still-valid non-compliant certificates were revoked

2025-03-11:

• 12:30 Updated CA application that includes a fix for the SCT issue was deployed

Related Incidents

Bug	Date	Description
(None)	N/A	N/A (No similar incidents having the same underlying cause(s) have been identified)

Root Cause Analysis

Contributing Factor 1: Programming Error

• Description: A programming error mistakenly applied timezone adjustment to the timestamps in the SCT extensions in the affected certificates, when no such adjustment was necessary. This affects the construction of SCT extensions for inclusion in final certificates. The code in question was not changed in subsequent releases of the CA application, until the issue was recently discovered and a fixed created and deployed.
• Timeline:
  • 2021-05-26 Release of CA application with certificate transparency support first deployed
• Detection: The programming error was identified by an internal code review. The error was not detected earlier partly because of factor 2 below.
• Interaction with other factors: The error in this factor was not detected for an extended period of time because of a lack of test case for the specific error in factor 2.

Contributing Factor 2: Lack of a test case for the specific error

• Description: We did not have a test case that verifies SCT signatures.
• Timeline:
  • 2021-05-20 Testing of the first release of CA application with certificate transparency support started
• Detection: The deficiency was identified during an internal review of why the issue was not uncovered in QA testing during previous CA application releases.
• Interaction with other factors: This factor contributes to the non-detection of the programming error in factor 1.

Lessons Learned

• What went well:

  • None

• What didn’t go well:

  • The programming error was not detected by linting tools, which contributed to its non-detection.

• Where we got lucky:

  • Practical impact on subscribers was minimal.

• Additional:

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Add test case to validate signatures in SCTs	Prevent	Root Causes #1 & #2	SCTs in certificates issued on or after 2025-03-11 have valid signatures	2025-05-02	Ongoing

Appendix

See attached CSV file.

Comment 4

[Nicol So]

Attachment: Listing of the affected certificates, in CSV format

Comment 5

[Andrew Ayer]

The certificates with the following SHA-256 hashes aren't published in any Certificate Transparency logs:

• 0f640db65453862fbd923a8d8a4909b5327172ff18a1477535627646728e581b
• 13d7fcf7dc0ecb89a4e500a3f112aa839a8535d636bfe209efa9159f378a112e
• 1581c2f5264c6bc7b8c480a2dd66c1463d6a055fa225cfcc0e1486cb1f9c8793
• 1a417d0aab40f471fb862a9cfdab4953f9af906b7ec357e299276685e1ce2852
• 1af9a1f444f45e02a2e4ee92d1132eccf979d668efa9fe3f5445ceaeb21b64cd
• 1f5fa2d2c5e075b08496918236960d16c39c861f770798a7d09d089e646fa363
• 5c1e8638559abc77d9f980ca26c0391df7a2b8ac0c0bbae9811f5647a2ee23ae
• 7104c515b64909fb822d7c16ad668fba3cfdb693b4a60bad137912f5f12f06e6
• 79583b4d2c444668df9a137b71a359f6f96d3491c468dd07748a211d1e477399
• 9598cb04ded047597e01ca0f42fc99b2b98a364075b65fc659ba35f03887dbd0
• aeca3a85baefcf093514a79df6f7ac66ec88af8becb54d1966264a741ddcea11
• b0d1caaa7281b0825d36622d7dbe783ab3e609bf577a31eb3c332615b3e55abc
• b3586779180832a4cfbf1abdf50ac90796b270367b2dede7ef6679273a5ef553
• b8870aa6abf89a7ef014c58270c19d917b212d9da041a962b10a00447d7e28e1
• c779654224770e38e7f03f7d56d8c6dd7c15fcb2e6b7767cf0548d542c6ae91c
• ce8dc1fb1a5e7199cbd8a9f1f75afa4af854651966d1c0557aa379224a41b4a6
• d00f11a60518830083fdf0f9749da47146e6bc9cf9da61b160abecda17cdf989

Could you please submit these certificates to a public CT log or attach them to this bug?

Comment 6

[Nicol So]

(In reply to Andrew Ayer from comment #5)

The certificates with the following SHA-256 hashes aren't published in any Certificate Transparency logs:

[17 certificate hashes, omitted here]

Could you please submit these certificates to a public CT log or attach them to this bug?

Thanks for the comment. Among the 17 certificate hashes you referenced, the corresponding certificates for all except three were already cataloged by crt.sh. (Actually, crt.sh also cataloged the corresponding precertificates, showing that they were published to CT logs.) I will upload information about these certificates and precertificates as an attachment.

For reasons unclear to me, three of the certificates were not cataloged by crt.sh. I will upload those certificates and the corresponding precertificates as an attachment.

Comment 7

[Nicol So]

Attachment: Certs referenced by AA already cataloged by crt.sh.csv

Comment 8

[Nicol So]

Attachment: Certs referenced by AA not cataloged by crt.sh.zip

Comment 9

[Andrew Ayer]

Thanks for the additional info. I just submitted the 17 missing certificates to CT logs. Note that crt.sh can sometimes contain certificates that aren't found in any CT logs, which was the case with the 14 certificates listed in Comment 7. As for precertificates, while usually they can serve as a stand-in for certificates, in this case the certificates need to be provided since the problem is in the SCT extension.

Comment 10

[Nicol So]

Update: We have completed the action item in comment 3 (adding test case to validate signatures in SCTs).

Comment 11

[Nicol So]

Report Closure Summary

• Incident description: CommScope discovered that the SCT extensions in certificates issued by its public CAs fail validation.

• Incident Root Cause(s): A programming error mistakenly applied an unnecessary timezone adjustment to the timestamps when constructing the SCT extensions in the affected certificates. We did not have a test case that verifies SCT signatures.

• Remediation description: The CA application involved was updated to correct the programming error. A test case was added to pre-deployment QA testing to verify the SCT signatures.

• Commitment summary: Add test case to validate signatures in SCTs.

The Action Item disclosed in this report has been completed as described, and we request its closure.

Comment 12

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-05-15.