Closed Bug 2006483 Opened 2 months ago Closed 1 month ago

IdenTrust: CT Logging Mistakes

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: roots, Assigned: roots)

Details

(Whiteboard: [ca-compliance] [uncategorized])

Attachments

(1 file)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36

Steps to reproduce:

IdenTrust: CT Logging Mistakes

Preliminary Incident Report

Summary

  • Incident Description
    Based on the public discussion, although our CA was not among those listed, we reviewed our Certificate Transparency (CT) log activity and confirmed a similar incident. One customer reported encountering the error net::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED after deploying TLS certificates issued by our CA. The root cause was the inclusion of a CT log with a “Qualified” status for 2027 rather than one marked as “Usable.” We have since remediated the issue by removing the “Qualified” log from our configuration and instructed the customer to replace the affected certificates.
  • Relevant policies:
    From Mozilla's definition of "misissuance" - A "misissuance" is defined as any certificate issued in contravention of any applicable standard, process or document - so it could be RFC non-compliant, BR non-compliant, issued contrary to the CA's CP/CPS, or have some other flaw or problem.
    Source of incident disclosure:
    CA Owner self-disclosed

Impact

This issue has affected at least one TLS Subscriber.
The root cause is under investigation, and a full incident report will be provided no later than December 30, 2025.

Assignee: nobody → roots
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000036

  • Incident description:

Based on this public discussion, although our CA was not among those listed, we reviewed our Certificate Transparency (CT) log activity and confirmed a similar incident. One customer reported encountering the error net::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED after deploying TLS certificates issued by our CA. The root cause was the inclusion of a CT log with a “Qualified” status for 2027 rather than one marked as “Usable.” We have since remediated the issue by removing the “Qualified” log from our configuration and instructed the customer to replace the affected certificates.

  • Timeline summary:

  • Non-compliance start date: 2025-12-02 17:40:30

  • Non-compliance end date: 2025-12-02 18:48:45

  • Non-compliance identified date: 2025-12-15 17:36

  • Relevant policies:
    From Mozilla's definition of "misissuance" - A "misissuance" is defined as any certificate issued in contravention of any applicable standard, process or document - so it could be RFC non-compliant, BR non-compliant, issued contrary to the CA's CP/CPS, or have some other flaw or problem.

  • Source of incident disclosure:
    CA Owner self-disclosed

Impact

  • Total number of certificates: 49

  • Total number of "remaining valid" certificates: 0

  • Affected certificate types: TLS

  • Incident heuristic: See attached csv file

  • Was issuance stopped in response to this incident, and why or why not?:
    Issuance remained unaffected because the CT log was cleared from the internal list within two hours.

  • Analysis:
    This issue impacted 46 internal certificates which were revoked on 2025-12-19. The remaining 3 certificates were revoked on 2025-12-30. We believe the 5-day revocation requirement under BR 4.9.1.1 does not apply in this case, as none of those circumstances occurred from our perspective in this case.

  • Additional considerations:

  • CA/B Forum Baseline Requirements do not restrict the use of ‘Qualified’ logs; they only require that precertificates be logged in at least two public logs.

  • Chrome’s CT policy allows ‘Qualified’; however, certificates using them may not work in older Chrome versions.

  • CCADB guidelines also do not indicate this is an incident.

We are disclosing this because some certificates may not properly validate in all versions of Chrome browsers.

Timeline

All times are in UTC
2025-12-02 17:40:30 Enabled ‘Argon2027h1’ (qualified CT log) in our systems
2025-12-02 18:00:00 Customer reported encountering the error net::ERR_CERTIFICATE_TRANSPARENCY_REQUIRE
2025-12-02 18:48:45 Removed ‘Argon2027h1’ from our systems
2025-12-11: 14:56 – Noticed list of CA’s with CT logging Mistakes posted via this webpage
2025-12-15 17:36 Agreed issue should be reported
2025-12-16: Disclosed preliminary incident
2025-12-19 19:01 Revoked 46 internal certificates and instructed affected customer to revoke
2025-12-3019:15:12 Revoked the 3 remaining customer certificates

Related Incidents

Bug Date Description
2005939 2025-12-14 CT logging mistakes

Root Cause Analysis

Contributing Factor #: Enabling Qualified Logs

  • Description:
    We enabled this 'Qualified’ log, acknowledging that Chrome’s policy states: "Qualified logs may not be recognized in out-of-date versions of Chrome." We've refined our internal process to ensure a thorough review before enabling ‘Qualified" logs.

Lessons Learned

  • What went well:
    Resolution occurred within two hours.

  • What didn’t go well:
    We overlooked the potential compatibility risk when enabling ‘Qualified’ logs, which could impact Chrome browser recognition.

  • Where we got lucky:
    Most of the certificates were internal to IdenTrust

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Remove Qualified logs from CT log disclosure list Correct Root Cause Successful validation Criteria 2025-02-02 Completed

Appendix

Details of affected certificates

See attached csv file

Whiteboard: [ca-compliance] [uncategorized]

Report Closure Summary

  • Incident description:
    Inclusion of a CT log with a “Qualified” status for 2027 rather than one marked as “Usable.” Which caused some customers to get this error: net::ERR_CERTIFICATE_TRANSPARENCY_REQUIRED”.
    We have since remediated the issue by removing the “Qualified” log from our configuration and instructed the customer to replace the affected certificates.
  • Incident Root Cause(s):
    Enabled 'Qualified log, acknowledging that Chrome’s policy states: "Qualified logs may not be recognized in out-of-date versions of Chrome."
  • Remediation description:
    Removed the Qualified log and updated internal process to ensure a thorough review before enabling Qualified logs.

Commitment summary:
All Action Items disclosed in this report have been completed as described, and we request its closure.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2026-01-20.

Whiteboard: [ca-compliance] [uncategorized] → [close on 2026-01-20] [ca-compliance] [uncategorized]
Flags: needinfo?(incident-reporting)

updated internal process to ensure a thorough review before enabling Qualified logs.

Could you expand on what this review will entail? Under what circumstances would using a Qualified log be justified?

Flags: needinfo?(roots)

(In reply to Andrew Ayer from comment #5)

updated internal process to ensure a thorough review before enabling Qualified logs.
Could you expand on what this review will entail? Under what circumstances would using a Qualified log be justified?

The review will consider scenarios in which disclosing to a Qualified log may be appropriate. Specifically, we would like to retain the option to use Qualified logs when diversity of log operators is needed but there are limited “usable” logs available. In such cases, including a Qualified log can help satisfy multi log SCT requirements across different operators, particularly when the log is expected to achieve usable status soon.

Additionally, during coordination with a new log operator, we may begin submitting precertificates to their Qualified log as part of early adoption testing. This allows our issuance workflows, monitoring systems, and audit processes to be validated and ready once the log transitions to usable or read only status.

Flags: needinfo?(roots)
Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Flags: needinfo?(incident-reporting)
Resolution: --- → FIXED
Whiteboard: [close on 2026-01-20] [ca-compliance] [uncategorized] → [ca-compliance] [uncategorized]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: