Open Bug 2055551 Opened 1 month ago Updated 4 hours ago

HARICA: Issuance of Server TLS Certificates with id-kp-clientAuth KeyPurposeID against CP/CPS

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: public-incident-reports, Assigned: public-incident-reports, NeedInfo)

Details

(Whiteboard: [ca-compliance] [__-misissuance] Next update 2026-09-18)

Attachments

(1 file)

clientAuth_appendix.md
5.55 MB, text/plain
Details

Preliminary Incident Report

Summary

  • Incident description: HARICA routinely updates its CP/CPS to reflect industry policy changes. On 2025-02-15, Chrome's Root Store Policy v1.6 required the removal of the id-kp-clientAuth KeyPurposeID from the extKeyUsage extension, effective 2026-06-15. We updated HARICA CP/CPS v4.9 on 2025-03-14 to incorporate this deadline. While Chrome subsequently extended this deprecation date to 2027-03-15 in version 1.8 of their policy, HARICA failed to update its CP/CPS to reflect the extension. Consequently, we incorrectly continued to allow the id-kp-clientAuth KeyPurposeID in TLS Certificates after our internal 2026-06-15 CP/CPS cutoff.

All TLS Certificates issued after 2026-06-15 00:00:01 (UTC) are affected and will be replaced and revoked within 5 days.

The affected certificate profiles were updated to remove the id-kp-clientAuth EKU temporarily until an updated CP/CPS was released.

A full incident report will be posted no later than 2026-07-30.

We thank the reporter for precisely identifying the problem and providing the relevant normative references, which enabled our team to rapidly confirm the finding.

  • Relevant policies: Section 7.1.2.3 of the HARICA CP/CPS
  • Source of incident disclosure: Third-party reported

Thank you for the preliminary report.

Because this incident appears to concern a divergence between HARICA’s CP/CPS and its issuance systems that persisted for approximately one month and was detected by a third party, these questions intend to focus on the structural controls that we think should have prevented or, at least detected the misissuance sooner. Please incorporate the responses to these questions directly into HARICA's Full Incident Report (i.e., we are explicitly waiving the CCADB Incident Reporting Guidelines expectation that these are answered in 7 days).

If per-profile CP/CPS conformance controls existed and functioned as desired, this incident could not have occurred as described. For each control HARICA describes in response to the following questions, please state whether it existed and was operating before 2026-06-15, and if so, why it did not detect the delta subject of this report.

We will also emphasize that responses proposing additional training, additional human review, or enhanced monitoring of upstream policy communications will be considered non-responsive and ultimately ineffective at addressing the root cause of this incident. Where claims can be tied to verifiable artifacts, CP/CPS version history, CCADB records, crt.sh, please do so.

Questions

Q1: Does HARICA maintain custom lints, specific to each of its issuing CAs and each certificate profile issued by those CAs, that evaluate conformance against HARICA’s own CP/CPS, covering every field and extension? Our use of “profile” should not strictly be interpreted as only the content in Section 7 of HARICA’s CPS, but should also consider the CA system specific configuration(s) responsible for the contents of issued certificates.

This is distinct from linting against the Baseline Requirements (such as PKIMetal or ZLint) or root program requirements. These custom lints would concern automated detection of deltas between:

  • (a) what the CP/CPS commits HARICA to,
  • (b) what the CA systems are configured to produce, and
  • (c) what issued certificates actually contain.

Note: Neither existing public linters nor human review can fill this gap. Public linters have no knowledge of commitments unique to HARICA’s CP/CPS (as highlighted in this incident, the prohibited EKU was fully BR-compliant). Human review demonstrably failed here, if the CP/CPS change was authored by humans, the profile configured by humans, and the delta went unnoticed for a month. If no automated CP/CPS conformance control exists today, please state so directly, help us understand why they do not exist, and detail what will fill this gap in the Action Items section of the Full Incident Report.

Q2: If such custom lints exist, please provide, per active certificate profile (also considering CA system configuration profiles, not just those described in HARICA’s CPS):

  • (a) The date each profile was last evaluated against the then-current CP/CPS.
  • (b) The specific mechanism that performed the evaluation.
  • (c) Whether these controls run pre-issuance as a blocking control, post-issuance on a defined cadence, or both.
  • (d) Total coverage expressed as [profiles covered] / [total active profiles].

If this information cannot be produced or disclosed publicly, please state so explicitly rather than describing controls in general terms.

Q3: What specific event triggers review and updates of these controls? In this incident, the trigger was a CP/CPS policy change (v4.9 introducing the 2026-06-15 cutoff). Was any technical process expected to translate that policy change into profile configuration updates and corresponding automated lint rules? If yes, identify where that process failed. If no, explain why a CP/CPS change is not considered a required trigger for updating conformance tooling.

Q4: Does HARICA maintain positive and negative test cases for its conformance controls? Please demonstrate the control that now catches this specific failure by confirming that an existing test certificate matching the affected profile containing id-kp-clientAuth with a notBefore date after 2026-06-15 - but before 2026-07-17 - fails validation today. Please state the exact date this control entered production. If that date is after this incident was reported, please state so plainly. Given recent changes made to HARICA’s CP/CPS (extending this original date to 2027-03-15), please describe a separate lint’s adoption considering that timeline.

Additional Comments:

C1: The preliminary report identifies a single delta against a single section, 7.1.2.3.

Upon performing our own quick review of Section 7.1.2.3, we identified what appears to be additional non-compliance that, if confirmed by HARICA, should be subject of a separate incident report. This leads us to believe HARICA’s initial remedy of this bug, which has at least included a July 16 CPS update, did not cast a wide enough net, or the process used to evaluate certificate conformance remains incomplete. Further review may identify additional non-compliance.

The current version (4.13 at the time of this post) of HARICA’s CP/CPS states (emphasis ours):

authorityInformationAccess

For TLS and Qualified eSignature/eSeal Certificates, this extension SHALL be
present. For Client Authentication, Code Signing, S/MIME and Time-stamping
Certificates, this extension MAY be present. For Short-term Certificates used for
eSignatures and eSeals, this extension MAY NOT be present.

If present, it SHALL NOT be marked critical, and it SHALL contain the HTTP
URL of the Issuing CA’s OCSP responder (accessMethod = 1.3.6.1.5.5.7.48.1).

It SHOULD also contain the HTTP URL of the Issuing CA’s certificate
(accessMethod = 1.3.6.1.5.5.7.48.2).

This same language appears to have existed since at least Version 4.5 (March 2022).

We interpret it to mean:

  • TLS certificates will always have the AIA extension.
  • The AIA extension will always include the OCSP pointer.

This contradicts HARICA’s public announcement related to sunsetting OCSP, and what we observe in time-valid, unrevoked certificates (example, currently being served at https://www.harica.gr/en/, lacking an OCSP pointer).

C2: Specific to root cause, this incident was detectable on 2026-06-15 by automated means and preventable if CP/CPS changes were mechanically propagated into profile configuration and conformance checks. We would ask that the full report treat the absence of such controls as the root cause, or minimally as a major contributing factor. The missed Chrome Root Program policy update is the occasion on which that absence became visible, not the cause. The full report should identify the specific automated control being introduced, its coverage, its cadence, and its production date.

C3: We consider this (and the OCSP issue) as being 100% avoidable. From our view, a major lesson of this incident is not that revocation obligations for CP/CPS violations are too rigid (which has been an ongoing conversation since before the Chrome Root Program even existed). It is that CP/CPS conformance controls are demonstrably immature, and the community has not adequately addressed this gap, despite numerous instances of events that should have triggered such work. The community should expect responses to failures of this kind to take the form of more reliable conformance engineering, not advocacy for weaker obligations.

Flags: needinfo?(public-incident-reports)
Assignee: nobody → public-incident-reports
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [__-misissuance]

This is to acknowledge receipt of Chrome's message and to confirm HARICA will complete the root cause analysis and provide the Full Incident Report by 2026-07-24.

In the meantime, we have read and confirmed the issue with the CP/CPS language regarding the AIA OCSP URI. Until a CP/CPS update, we have halted issuance and we are working to fix the misconfiguration. A bug will be registered for this issue within 24 hours.

Please note that http://crl.harica.gr/HARICA-GEANT-TLS-R1.crl now revoked some(?) certificates

$ md5sum HARICA-GEANT-TLS-R1.crl
6e027d61b42bcd3e9dc3c8d078f3bea5  HARICA-GEANT-TLS-R1.crl

At least one affected certificate was in there and revoked yesterday:

$ openssl crl -inform DER -in HARICA-GEANT-TLS-R1.crl -noout -text | grep -A3 5F67129748580AF43932D6D1321CBF29
    Serial Number: 5F67129748580AF43932D6D1321CBF29
        Revocation Date: Jul 20 10:08:09 2026 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code:

Draft Incident Report

Sections 1–4 (Summary, Impact, Timeline, Related Incidents) and 8 (Appendix) are complete. Notably, section 4 details our review of 13 related incidents across 9 CA Owners reported in the last 5 years, including the common failure modes we identified and the solutions proposed therein. The Root Cause Analysis, Lessons Learned, and Action Items sections remain under analysis; we will post final content for them by 2026-07-30.

We are providing this update today to honor the commitment we made in Comment #2. That initial 7-day timeline was based on a misreading of the Chrome Root Program's request in Comment #1, which actually allowed 14 days for the response to their questions as part of the Full Incident Report.

By reconstructing the events leading to this incident, analyzing similar historical bugs, and addressing the Chrome Root Program's inquiries, we have now gathered extensive data. We will utilize the remaining time to synthesize this information and formulate actionable, comprehensive results.

Summary

  • CA Owner CCADB unique ID: A000035

  • Incident description: HARICA issued Server TLS Certificates containing the id-kp-clientAuth KeyPurposeID in the extKeyUsage extension after the removal deadline stated in its own CP/CPS. This occurred because HARICA's CP/CPS reflected the outdated deprecation deadline (2026-06-15) that no longer matched the current Chrome Root Program Policy (2027-03-15). The issuance was in compliance with Chrome's current policy, but in violation of the CP/CPS.

  • Timeline summary:

    • Non-compliance start date: 2026-06-15

    • Non-compliance identified date: 2026-07-15

    • Non-compliance end date: 2026-07-15

  • Relevant policies: HARICA CP/CPS 4.11 section 7.1.2.3

  • Source of incident disclosure: Third-party reported

Impact

  • Total number of certificates: 66105, of which 64949 non-expired, non-revoked at the time when the incident’s investigation started

  • Total number of "remaining valid" certificates: 0

  • Affected certificate types: Server TLS Certificates (DV, OV, EV)

  • Incident heuristic: The full corpus of affected certificates is disclosed in the Appendix.

  • Was issuance stopped in response to this incident, and why or why not?: Issuance was briefly stopped until the affected server TLS Certificate Profiles were updated to include the clientAuth EKU.

  • Analysis: N/A

  • Additional considerations: N/A

Timeline

All timestamps are in EEST (GMT+3) except where indicated otherwise.

  • 2025-02-15 Chrome's Root Store Policy v1.6 required the removal of the id-kp-clientAuth KeyPurposeID from the extKeyUsage extension, effective 2026-06-15 for existing CAs.

  • 2025-03-14 HARICA’s compliance team notifies internal teams about the upcoming removal of the clientAuth EKU on TLS Certificates and updates its CP/CPS v4.9 to incorporate this Root Store Policy update and the corresponding deadline.

  • 2025-07-15 Chrome's Root Store Policy v1.7 is released maintaining the same deadline (2026-06-15).

  • 2025-12-05 The CA team prepares the implementation tickets setting the “due date” before the 2026-06-15 cutoff date.

  • 2026-01-29 The Content team prepares a public announcement setting the “due date” before the 2026-06-15 cutoff date.

  • 2026-02-05 Chrome's Root Store Policy v1.8 is released pushing the cutoff date to 2027-03-15.

  • 2026-02-06 HARICA’s compliance team notifies internal teams about the changes introduced in version 1.8. The CA team updates the “due date”. As a sidenote, an additional ticket is created for CA certificates, with the original deadline (2026-06-15).

  • 2026-02-17 The Content team updates the public announcement postponing the change by approximately 9 months. The CA Engineering team updates the corresponding internal ticket due dates accordingly.

  • 2026-06-15: 00:00:01 (UTC) The TLS profiles documented in the CP/CPS required the removal of the clientAuth EKU, but the actual profiles continued to include it. (NON-COMPLIANCE BEGINS)

  • 2026-07-15: 15:07 CPR received notifying about the inconsistency between CP/CPS and actual TLS Certificates.

  • 2026-07-15: 15:10 Compliance team is notified, confirms the issue and issues an order for an emergency change of the CA profiles to include the clientAuth EKU to the TLS profiles, and of the CP/CPS to address the issue.

  • 2026-07-15: 15:30 Mass-revocation plan preparations begin.

  • 2026-07-15: 17:15 CA Engineering team stops issuance until the profiles are updated.

  • 2026-07-15: 17:19 TLS Certificate profiles are updated to include the clientAuth EKU. (NON-COMPLIANCE ENDS)

  • 2026-07-15: 17:21 CA Engineering team resumes issuance.

  • 2026-07-15: 17:59 Mass-revocation plan is approved to begin

  • 2026-07-16: 10:45 Internal coordination meeting on mass revocation to ensure all teams are clear on their required tasks.

  • 2026-07-16: 10:48 CP/CPS v4.13 is published with corrected language for the clientAuth EKU.

  • 2026-07-16: 12:05 CA Engineering team enables the clientAuth EKU to the server TLS certificate profiles.

  • 2026-07-16: 14:58 Additional testing and fine-tuning of ACME ARI to ensure re-issuance works as expected for the affected customers.

  • 2026-07-16: 15:48 Email notifications are sent to the affected subscribers.

  • 2026-07-16: 18:33 Bugzilla incident 2055551 (this bug) is created and a preliminary incident report is submitted.

  • 2026-07-17: 12:38 Verification that all non-ACME affected certificates can re-use the existing Domain and Identity validations.

  • 2026-07-17: 12:50 ARI information is updated and ACME clients supporting the ARI extension become aware of the need to replace certificates until 2026-07-19 23:59:59 UTC.

  • 2026-07-17: 17:26 Non-ACME replacement certificate issuance process begins and Subscribers notified.

  • 2026-07-17: 23:29 Verification that non-ACME replacement certificates were issued. Issuances that didn’t go through due to misconfigurations by the customers (e.g. CAA issues, use of compromised keys) were sent to the Support team for manual notification.

  • 2026-07-20: 12:55 Revocation of affected certificates starts.

  • 2026-07-20: 13:12 Revocation of 63525 certificates completed. The remaining 1424 out of the total 64949 non-expired non-revoked certificates when the incident was reported were either expired or revoked.

  • 2026-07-20 to 2026-07-24 Collection of the events that led to this incident, research and review of relevant incidents in Bugzilla, compilation of this report. The root cause analysis and analysis of potential remediation actions in progress.

Related Incidents

Based on our long-standing monitoring of bugs and discussions in the community, and the analysis made by the Chrome team with c#1, the problem of mis-issuances due to violations of a CA's own CP/CPS without a direct violation of a CA/Browser Forum requirement, is a well-known challenge in the community.

For this reason, we conducted a thorough search and review of this type of incidents to identify root causes and patterns. Our analysis also examined the remediation measures adopted in these cases, to develop a deeper understanding of the status and best practices in addressing this issue.

The results of this review are summarized in the following table (most recently-opened first) and serve as a basis for the incident analysis and the selection of remediation measures.

Bug Date Description
2056668 2026-09-21 HARICA: Issuance of Server TLS Certificates without AIA OCSP URI against CP/CPS Reported by a third party during the discussion of this incident (see Chrome’s additional comment C1 in c#1). The incident is at the preliminary report stage.
2056087 2026-09-17 Disig: CP/CPS misstatement regarding Key Usage criticality for TLS certificates Reported by a third party. The RCA called out a 2017 erroneous update of the CPS regarding KeyUsage criticality followed by a failure of the manual documentation review process to identify and address the issue when the TLS BR v2.0.0 introduced mandatory keyUsage criticality in 2023. Disig also identified a lack of automated cross-validation between human-readable policy documentation and active machine-readable issuance profiles as a contributing factor. Remediation action includes the implementation of automated / mandatory policy-to-profile linting matrix into the CP/CPS release process.
2048444 2026-06-17 IdenTrust: Issuance of TLS certificates with additional certificate policy OIDs, for which the CP/CPS had a date for removal (due to a missed Chrome Root Program policy update). Self-detected through an internal review of the CPS. The “Related Incidents” section focused on violations of own CPS. Remediation included process improvements, without any tooling updates or automation.
2011865 2026-01-22 TrustAsia: Issuance of DV SSL certificates containing IP addresses against CP/CPS – referenced by 2048444 Self-detected during the investigation of another incident (2011713). The RCA called out a failure to fully assess CP/CPS constraints when responding to business requirement changes, along with a gap in the pre-issuance linting tools in performing customized checks against stricter CP/CPS provisions. Remediation actions include establishing a mandatory checklist in the configuration change process to check against both the BRs and the CP/CPS and developing certificate linting tools for CP/CPS requirements for use pre-issuance.
1981680 2025-08-07 TunTrust: Issuance of an SSL OV certificate with the “emailAddress” attribute in the subjectDN without being allowed in the CP/CPS – referenced by 2048444 Self-detected during an internal audit. The RCA called out insufficient checks after changes, as the focus of the manual review is only on modifications, rather than the full profile. With regards to detection, the RCA called out the lack of post-issuance review of the certificate content against the subscriber profile defined in the CP/CPS. Remediation measures included updating the certificate profile management procedure to check all the certificate profile in the change control step, and updating the validation procedure to formally check the content of the certificate (post-issuance).
1979475 2025-07-25 Microsoft PKI Services: Issuance of TLS certificates without the Basic Constraints extension, which the CP/CPS specifies as mandatory. Self-detected through a CPS review (part of the enhanced monitoring response to incident 1962829, see below). The RCA identified the misalignment between linting tools and CPS enforcement as a contributing factor. Remediation included the implementation of internal linting rules that enforce CPS-specific requirements to supplement external BR-based linting tools. The relevant action item took the longest to complete (5 months).
1962830 2025-04-25 Microsoft PKI services: Issuance of TLS certificates without the OCSP URI, prior to the publication of the updated CP/CPS that makes it optional. Reported by a third party through a CPR. Remediation included process improvements (updating the project plan template to include CPS conflict checks prior to CA changes), without any tooling updates or automation.
1962829 2025-04-25 Microsoft PKI services: Introduction of a CPS typographical mistake (regarding keyEncipherment) that did not align with the unaltered practice. Reported by a third party through a CPR. The RCA pointed out the limitations of human review when performing document reviews, especially upon multiple/major changes or re-structuring. The initial remediation actions included an increase in the number of reviewers, plus migration to Git. Several issues relevant to our case were raised by the Chrome team (c#8), including the suggestion to translate its CP and CPS into custom lints to be incorporated in an already in-use linting solution. Digicert suggested (c#15) the use of mandatory machine-readable profiles, combined with automated profile generation and enforcement. Remediation actions include a CPS-to-lint translation based on a comprehensive review of linting rules against certificate profiles defined in the CP/CPS. The action item took 6.5 months more than initially projected. No details about the exact fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.
1921573 2024-09-27 Let’s Encrypt: No meaningful Subject DN – conflict between two sections of the CPS – also referenced by 1962829 Self-detected through a quarterly CP/CPS review. The RCA pointed out a de-sync between section 3.1.2 and section 7.1, after the latter was updated ahead of making corresponding changes to the CA behavior (Boulder) to allow empty CN. Remediation focused on an ad-hoc CP/CPS review specifically looking for any statements about Certificate contents outside of Section 7. Nothing related to process improvements or automation.
1890896 2024-04-10 Entrust: CPS typographical error - Unintentional addition of the policyQualifier in the CP/CPS OV certificate profile. – also referenced by 1962829 Self-detected through a CPS review amid the investigation of other incidents. The RCA indicated that amid multiple CPS fixes caused by concurrent incidents, a typographical a change was made to a generic certificate profile that applies to all TLS cert types (OV, EV). Remediation included process improvements (review CPS update procedure), without any tooling updates or automation.
1886876 2024-03-21 Let's Encrypt: keyCompromise key blocking deviation from CP/CPS Self-detected through a quarterly CP/CPS review (the process is described in c#4). The RCA called out the lack of effective mechanisms for ensuring that changes to CA behavior which would violate the CPS are preceded by changes to the CPS to allow said behavior. The approach is based on the concept of changes flowing into the CPS from the CA, rather than the other way around. Remediation actions include establishing automation requiring confined CP/CPS review with each proposed code change that introduces a new feature flag (hence, smaller targeted reviews at shorter intervals).
1715455 2021-06-09 Let's Encrypt: certificate lifetimes 90 days plus one second. Reported by a third party. The RCA pointed out the non-compliance caused due to updating the CPS to stricter than the BR certificate maximum validity requirement, without identifying the need to change the implementation, and without the human-drive initial and any subsequent CPS reviews catching the issue. Remediation actions addressed the issue at hand, without a systemic change.
1708965 2021-05-02 KIR S.A.: Certificates issued with validity period 1 second greater than stated in own CPS (1 year). Reported by a third party. According to the RCA, due to a failure to consider the validity period definition of the RFC 5280, there was a mismatch between the practice (issuances) and the policy. Remediation, part of a related incident (1705904), included process improvements (dual person review of CPS updates), plus an ad-hoc CPS review by an external auditor.

Root Cause Analysis

In progress; will be included in the next update.

Lessons Learned

In progress; will be included in the next update.

Action Items

In progress; will be included in the next update.

Appendix

Attached the population of affected certificates – including at-the-time active, expired and previously revoked – in the format required by CCADB’s Incident Reporting Guidelines.

Attached file clientAuth_appendix.md —

Correction: The first two bugs in the Related Incidents were created in July, not September.

In Comment #4 we committed to posting the final Root Cause Analysis, Lessons Learned and Action Items by 2026-07-30.

We need one additional day and will post them by 2026-07-31, for two reasons. Our review of the CP/CPS version history surfaced further detail that we would rather reflect accurately than post and revise shortly afterwards. We are also working on bug 2056668 in parallel, and the two incidents appear to share much of their root cause and remediation. We would rather take the extra day and be confident that what we conclude here holds for both than present two analyses that do not sit together.

The responses to the Chrome Root Program's questions in Comment #1 are incorporated into the report and will be posted with it. We appreciate the community's patience.

Even if you were requesting some extra time that isn't anything the community or root program can just approve, or apply retroactively in this case. A Full incident report MUST be provided within 14 days as per CCADB policy. You are already hours past the threshold even timing off of the start of this bug report.

Missing the 14 days MUST threshold will be another incident, which is not down to the preference of a CA. If the incidents share root causes and remediations, that will only mean it's easier for you to make the 14 day threshold in bug #2056668 in a timely manner.

If there was any awareness of the schedule being impossible prior to the deadline being broken then it should have been posted then. That would not change the result of no one being able to approve any extra time, however.

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000035

  • Incident description: HARICA issued Server TLS Certificates containing the id-kp-clientAuth KeyPurposeID in the extKeyUsage extension after the removal deadline stated in its own CP/CPS. This occurred because HARICA's CP/CPS reflected the outdated deprecation deadline (2026-06-15) that no longer matched the current Chrome Root Program Policy (2027-03-15). The issuance was in compliance with Chrome's current policy, but in violation of the CP/CPS.

  • Timeline summary:

    • Non-compliance start date: 2026-06-15

    • Non-compliance identified date: 2026-07-15

    • Non-compliance end date: 2026-07-15

  • Relevant policies: HARICA CP/CPS 4.11 and 4.12, section 7.1.2.3

  • Source of incident disclosure: Third-party reported

Impact

  • Total number of certificates: 66,105, of which 64,949 non-expired, non-revoked at the time when the incident's investigation started

  • Total number of "remaining valid" certificates: 0

  • Affected certificate types: Server TLS Certificates (DV, OV, EV)

  • Incident heuristic: The full corpus of affected certificates is disclosed in the Appendix.

  • Was issuance stopped in response to this incident, and why or why not?: Issuance was briefly stopped until the affected server TLS Certificate Profiles were updated to include the clientAuth EKU.

  • Analysis: N/A

  • Additional considerations: N/A

Timeline (updated)

All timestamps are in EEST (GMT+3) except where indicated otherwise.

  • 2025-02-15: Chrome's Root Store Policy v1.6 required the removal of the id-kp-clientAuth KeyPurposeID from the extKeyUsage extension, effective 2026-06-15 for existing CAs.

  • 2025-03-14: HARICA's compliance team notifies internal teams about the upcoming removal of the clientAuth EKU on TLS Certificates and updates its CP/CPS v4.9 to incorporate this Root Store Policy update and the corresponding deadline.

  • 2025-07-15: Chrome's Root Store Policy v1.7 is released maintaining the same deadline (2026-06-15).

  • 2025-12-05: The CA team prepares the implementation tickets setting the "due date" before the 2026-06-15 cutoff date.

  • 2026-01-29: The Content team prepares a public announcement setting the "due date" before the 2026-06-15 cutoff date.

  • 2026-02-05: Chrome's Root Store Policy v1.8 is released pushing the cutoff date to 2027-03-15.

  • 2026-02-06: HARICA's compliance team notifies internal teams about the changes introduced in version 1.8. The CA team updates the "due date". As a sidenote, an additional ticket is created for CA certificates, with the original deadline (2026-06-15).

  • 2026-02-17: The Content team updates the public announcement postponing the change by approximately 9 months. The CA Engineering team updates the corresponding internal ticket due dates accordingly.

  • 2026-06-15 00:00:01 (UTC): The TLS profiles documented in the CP/CPS required the removal of the clientAuth EKU, but the actual profiles continued to include it. (NON-COMPLIANCE BEGINS)

  • 2026-06-29 15:02: HARICA completes and submits its 2026 self-assessment, based on a comparison of the CP/CPS v4.11 (2025-11-28) against the CA/B Forum Requirements and Root Store Policies. The assessment uses CCADB self-assessment template version 1.8 (2026-02-26), which incorporates Chrome Root Store Policy v1.8, and HARICA responds to clauses DEDICATED-1 and DEDICATED-2, covering Dedicated TLS Server Authentication PKI Hierarchies, with "CURRENTLY COMPLY and INTEND TO CONTINUE COMPLIANCE". The responses were accurate: the applicable Chrome deadline is 2027-03-15. The divergence subject of this report is not detected.

  • 2026-07-10: CP/CPS v4.12 is published to incorporate Chrome's Root Store Policy v1.8; timeline is updated to 2027-03-15 in section 7.1.2.2 regarding CA certificates, but not in section 7.1.2.3 regarding leaf certificates.

  • 2026-07-15 15:07: CPR received notifying about the inconsistency between CP/CPS and actual TLS Certificates.

  • 2026-07-15 15:10: Compliance team is notified, confirms the issue and issues an order for an emergency change of the CA profiles to include the clientAuth EKU to the TLS profiles, and of the CP/CPS to address the issue.

  • 2026-07-15 15:30: Mass-revocation plan preparations begin.

  • 2026-07-15 17:15: CA Engineering team stops issuance until the profiles are updated.

  • 2026-07-15 17:19: TLS Certificate profiles are updated to include the clientAuth EKU. (NON-COMPLIANCE ENDS)

  • 2026-07-15 17:21: CA Engineering team resumes issuance.

  • 2026-07-15 17:59: Mass-revocation plan is approved to begin.

  • 2026-07-16 10:45: Internal coordination meeting on mass revocation to ensure all teams are clear on their required tasks.

  • 2026-07-16 10:48: CP/CPS v4.13 is published with corrected language for the clientAuth EKU.

  • 2026-07-16 12:05: CA Engineering team enables the clientAuth EKU to the server TLS certificate profiles.

  • 2026-07-16 14:58: Additional testing and fine-tuning of ACME ARI to ensure re-issuance works as expected for the affected customers.

  • 2026-07-16 15:48: Email notifications are sent to the affected subscribers.

  • 2026-07-16 18:33: Bugzilla incident 2055551 (this bug) is created and a preliminary incident report is submitted.

  • 2026-07-17 12:38: Verification that all non-ACME affected certificates can re-use the existing Domain and Identity validations.

  • 2026-07-17 12:50: ARI information is updated and ACME clients supporting the ARI extension become aware of the need to replace certificates until 2026-07-19 23:59:59 UTC.

  • 2026-07-17 17:26: Non-ACME replacement certificate issuance process begins and Subscribers notified.

  • 2026-07-17 23:29: Verification that non-ACME replacement certificates were issued. Issuances that didn't go through due to misconfigurations by the customers (e.g. CAA issues, use of compromised keys) were sent to the Support team for manual notification.

  • 2026-07-20 12:55: Revocation of affected certificates starts.

  • 2026-07-20 13:12: Revocation of 63,525 certificates completed. The remaining 1,424 out of the total 64,949 non-expired non-revoked certificates when the incident was reported were either expired or revoked.

  • 2026-07-20 to 2026-07-24: Collection of the events that led to this incident, research and review of relevant incidents in Bugzilla, compilation of the Draft Incident Report.

  • 2026-07-21: CP/CPS v4.14 is published, making the AIA OCSP URI optional and addressing the divergence reported in bug 2056668.

  • 2026-07-24: Submission of the Draft Incident Report in Bugzilla.

  • 2026-07-24 to 2026-07-31: Detailed Root Cause Analysis, Lessons Learned and Remediation Action Plan, and compilation of the Full Incident Report (this report).

Related Incidents

Based on our long-standing monitoring of bugs and discussions in the community, and the analysis made by the Chrome team with c#1, the problem of mis-issuances due to violations of a CA's own CP/CPS without a direct violation of a CA/Browser Forum requirement, is a well-known challenge in the community.

For this reason, we conducted a thorough search and review of this type of incidents to identify root causes and patterns. Our analysis also examined the remediation measures adopted in these cases, to develop a deeper understanding of the status and best practices in addressing this issue.

The results of this review are summarized in the following table (most recently-opened first) and serve as a basis for the incident analysis and the selection of remediation measures.

Bug Date Description
2056668 2026-07-21 HARICA: Issuance of Server TLS Certificates without AIA OCSP URI against CP/CPS. Reported by a third party during the discussion of this incident (see Chrome's additional comment C1 in c#1). The incident is at the preliminary report stage.
2056087 2026-07-17 Disig: CP/CPS misstatement regarding Key Usage criticality for TLS certificates. Reported by a third party. The RCA called out a 2017 erroneous update of the CPS regarding KeyUsage criticality followed by a failure of the manual documentation review process to identify and address the issue when the TLS BR v2.0.0 introduced mandatory keyUsage criticality in 2023. Disig also identified a lack of automated cross-validation between human-readable policy documentation and active machine-readable issuance profiles as a contributing factor. Remediation action includes the implementation of automated / mandatory policy-to-profile linting matrix into the CP/CPS release process.
2048444 2026-06-17 IdenTrust: Issuance of TLS certificates with additional certificate policy OIDs, for which the CP/CPS had a date for removal (due to a missed Chrome Root Program policy update). Self-detected through an internal review of the CPS. The "Related Incidents" section focused on violations of own CPS. Remediation included process improvements, without any tooling updates or automation.
2011865 2026-01-22 TrustAsia: Issuance of DV SSL certificates containing IP addresses against CP/CPS – referenced by 2048444. Self-detected during the investigation of another incident (2011713). The RCA called out a failure to fully assess CP/CPS constraints when responding to business requirement changes, along with a gap in the pre-issuance linting tools in performing customized checks against stricter CP/CPS provisions. Remediation actions include establishing a mandatory checklist in the configuration change process to check against both the BRs and the CP/CPS and developing certificate linting tools for CP/CPS requirements for use pre-issuance.
1981680 2025-08-07 TunTrust: Issuance of an SSL OV certificate with the "emailAddress" attribute in the subjectDN without being allowed in the CP/CPS – referenced by 2048444. Self-detected during an internal audit. The RCA called out insufficient checks after changes, as the focus of the manual review is only on modifications, rather than the full profile. With regards to detection, the RCA called out the lack of post-issuance review of the certificate content against the subscriber profile defined in the CP/CPS. Remediation measures included updating the certificate profile management procedure to check all the certificate profile in the change control step, and updating the validation procedure to formally check the content of the certificate (post-issuance).
1979475 2025-07-25 Microsoft PKI Services: Issuance of TLS certificates without the Basic Constraints extension, which the CP/CPS specifies as mandatory. Self-detected through a CPS review (part of the enhanced monitoring response to incident 1962829, see below). The RCA identified the misalignment between linting tools and CPS enforcement as a contributing factor. Remediation included the implementation of internal linting rules that enforce CPS-specific requirements to supplement external BR-based linting tools. The relevant action item took the longest to complete (5 months).
1962830 2025-04-25 Microsoft PKI services: Issuance of TLS certificates without the OCSP URI, prior to the publication of the updated CP/CPS that makes it optional. Reported by a third party through a CPR. Remediation included process improvements (updating the project plan template to include CPS conflict checks prior to CA changes), without any tooling updates or automation.
1962829 2025-04-25 Microsoft PKI services: Introduction of a CPS typographical mistake (regarding keyEncipherment) that did not align with the unaltered practice. Reported by a third party through a CPR. The RCA pointed out the limitations of human review when performing document reviews, especially upon multiple/major changes or re-structuring. The initial remediation actions included an increase in the number of reviewers, plus migration to Git. Several issues relevant to our case were raised by the Chrome team (c#8), including the suggestion to translate its CP and CPS into custom lints to be incorporated in an already in-use linting solution. Digicert suggested (c#15) the use of mandatory machine-readable profiles, combined with automated profile generation and enforcement. Remediation actions include a CPS-to-lint translation based on a comprehensive review of linting rules against certificate profiles defined in the CP/CPS. The action item took 6.5 months more than initially projected. No details about the exact fixes for any gaps identified during the comprehensive review of our linting and CP/CPS alignment.
1921573 2024-09-27 Let's Encrypt: No meaningful Subject DN – conflict between two sections of the CPS – also referenced by 1962829. Self-detected through a quarterly CP/CPS review. The RCA pointed out a de-sync between section 3.1.2 and section 7.1, after the latter was updated ahead of making corresponding changes to the CA behavior (Boulder) to allow empty CN. Remediation focused on an ad-hoc CP/CPS review specifically looking for any statements about Certificate contents outside of Section 7. Nothing related to process improvements or automation.
1890896 2024-04-10 Entrust: CPS typographical error - Unintentional addition of the policyQualifier in the CP/CPS OV certificate profile. – also referenced by 1962829. Self-detected through a CPS review amid the investigation of other incidents. The RCA indicated that amid multiple CPS fixes caused by concurrent incidents, a typographical change was made to a generic certificate profile that applies to all TLS cert types (OV, EV). Remediation included process improvements (review CPS update procedure), without any tooling updates or automation.
1886876 2024-03-21 Let's Encrypt: keyCompromise key blocking deviation from CP/CPS. Self-detected through a quarterly CP/CPS review (the process is described in c#4). The RCA called out the lack of effective mechanisms for ensuring that changes to CA behavior which would violate the CPS are preceded by changes to the CPS to allow said behavior. The approach is based on the concept of changes flowing into the CPS from the CA, rather than the other way around. Remediation actions include establishing automation requiring confined CP/CPS review with each proposed code change that introduces a new feature flag (hence, smaller targeted reviews at shorter intervals).
1715455 2021-06-09 Let's Encrypt: certificate lifetimes 90 days plus one second. Reported by a third party. The RCA pointed out the non-compliance caused due to updating the CPS to stricter than the BR certificate maximum validity requirement, without identifying the need to change the implementation, and without the human-driven initial and any subsequent CPS reviews catching the issue. Remediation actions addressed the issue at hand, without a systemic change.
1708965 2021-05-02 KIR S.A.: Certificates issued with validity period 1 second greater than stated in own CPS (1 year). Reported by a third party. According to the RCA, due to a failure to consider the validity period definition of the RFC 5280, there was a mismatch between the practice (issuances) and the policy. Remediation, part of a related incident (1705904), included process improvements (dual person review of CPS updates), plus an ad-hoc CPS review by an external auditor.

Root Cause Analysis

We worked back from the issuance using a 5-Whys chain to identify contributing factors and reveal the principal root cause. Chrome's questions Q1 to Q4 are answered in CF-2 and CF-3; if anything is missing, we will be happy to follow up with additional information.

In short: the certificate profiles kept the clientAuth EKU, correctly under Chrome Root Store Policy v1.8, while the CP/CPS still carried the earlier 2026-06-15 cutoff. Initially, the CP/CPS was not updated because a policy relaxation (an effective date moved to later) was treated as needing no action. Nothing caught it afterwards, because our CP/CPS reviews check the document against external requirements, and a stricter clause passes. When the CP/CPS was finally updated for v1.8, the date update was applied to section 7.1.2.2 (ICA profiles) instead of section 7.1.2.3 (leaf certificate profiles) and the internal review did not identify the omission. No automated control checks certificates against our own CP/CPS, and none can be derived from it in its current format and content. Underneath all of this, the certificate profile exists in two places at once, in the CP/CPS and in the CA configuration, with a manual process keeping them in sync, while the CP/CPS copy is binding on issuance.

We identify CF-4 as the principal root cause. CF-1 to CF-3 are the conditions that let it produce this incident and kept it invisible for five months.

We reviewed thirteen comparable incidents across nine CA Owners, summarized in the Related Incidents section. The pattern is consistent. The CP/CPS and the issuance systems drift apart, in either direction: the document is left behind by a Root Program change (2048444), or practice moves ahead of the document (1962830), or a business change is made without checking it against the CP/CPS (2011865). Human review does not catch it, whether the defect is a typographical error introduced while reformatting (1962829), an edit applied to the wrong profile (1890896), or two sections of the same document falling out of step (1921573). Remediation is usually aimed at the coordination step that failed rather than at the duplication that made the failure possible, through process improvements without tooling changes (2048444, 1962830, 1890896). Where CAs have gone further, the direction has been to derive lints from the CP/CPS rather than maintain them separately (1962829, 1979475, 2056087), or to require a CP/CPS review automatically whenever the CA's behavior changes (1886876). That review informed the factors below and shaped the direction of our remediation actions.

Contributing Factor #1 (CF-1): A Root Program relaxation was treated as needing no CP/CPS action

  • Description: Each incoming external change is assigned to a compliance officer, who passes it to the affected teams and, where needed, raises a CP/CPS update. For Chrome v1.6 all three legs ran (CA configuration, communications, CPS update), and the CPS update was published as v4.9. For v1.8 only two ran: the CA team moved the profile due date and kept 2026-06-15 for CA certificates in a separate ticket, and the content team revised the public notice. No CP/CPS update was raised, because a relaxation was assumed to need no action. That overlooked the fact that the document already carried 2026-06-15 as a commitment of our own. Staying stricter than Chrome is safe as far as Chrome is concerned, but our own CP/CPS binds issuance regardless of what the ecosystem allows. The change was also handled as a continuation of the v1.6 work rather than a new item to analyze. That fitted the two teams with open items to adjust; compliance had none, its v1.6 work having been published and closed in March 2025. Canceling a pending order needs no rollback, but a published document does.

  • Timeline:

    • 2025-02-15: Chrome v1.6 requires removal of the EKU by 2026-06-15.

    • 2025-03-14: CP/CPS v4.9 published with that deadline.

    • 2026-02-05: Chrome v1.8 moves the cutoff to 2027-03-15.

    • 2026-02-06: Internal teams notified; the CA team updates the due date and opens a separate ticket for CA certificates retaining 2026-06-15; no CP/CPS action raised.

    • 2026-02-17: The Content team updates the public announcement; CA Engineering updates the corresponding internal ticket due dates.

    • 2026-06-15 00:00:01 (UTC): Non-compliance begins.

    • 2026-07-10: CP/CPS v4.12 is published to incorporate Chrome Root Store Policy v1.8; the date is updated in section 7.1.2.2 and not in section 7.1.2.3 (see CF-2).

  • Detection: Not detected internally. Reported by a third party on 2026-07-15 at 15:07 and confirmed three minutes later. The compliance officer assigned to each change checks that the notified teams have done their part, but nothing recorded that the CP/CPS had been considered and judged not to need a change, so there was nothing to check.

  • Interaction with other factors: This created the divergence; CF-2 and CF-3 let it run. The assumption itself follows from CF-4: a relaxation makes no difference to policy text, but it is decisive for a specification that binds issuance on a date, and nothing in how the CP/CPS is written or reviewed marks that difference.

  • Root Cause Analysis methodology used: 5-Whys

Contributing Factor #2 (CF-2): CP/CPS reviews check the document only against external requirements, and did not catch the v4.12 update landing in the wrong section

  • Description: Two review controls apply. Every change is reviewed under a four-eyes principle and approved by the PMC. Separately, we review the CP/CPS at least annually as part of the CCADB self-assessment, a clause-by-clause comparison against the CA/B Forum Requirements and the Root Store Policies. Both run in the same direction: they ask whether the CP/CPS falls short of what is required externally. Neither asks whether it commits us to more than our systems do, and a stricter CP/CPS breaks no external requirement and passes. Finding this would have meant comparing Section 7 against the profile configuration, or against certificates as issued, which neither control does.

    The four-eyes review has a second limitation. The second reviewer is not usually a compliance officer with a broad view of incoming normative changes, so they review the text as drafted rather than checking where else the requirement applies. Scope is therefore set by the drafter and carried through unchanged. That showed in v4.12, published 2026-07-10 to bring in Chrome v1.8: the deadline appears in section 7.1.2.2 for ICA profiles and in section 7.1.2.3 for leaf certificate profiles, and the update was applied to the wrong one. Section 7.1.2.2 was revised and section 7.1.2.3, where the superseded date stood, was not. The review did not catch it. Had the update gone to the right section, we would have found the problem ourselves and ended it 25 days in, rather than having it reported to us five days later. Separately, the resulting text in 7.1.2.2 does not breach any Root Program requirement or our practice; it remains uncorrected as at the date of this report and will be addressed in a forthcoming CP/CPS update.

    This answers Q3 in part. A CP/CPS change does trigger downstream work: changes to Section 7 are passed to the CA team and implemented in the profiles. What is missing is a technical control enforcing it, i.e. a mechanism to prevent a change to Section 7 from becoming effective without the corresponding profile change, and a mechanism to check afterwards that the two agree. The link between the document and the systems is procedural throughout.

  • Timeline:

    • 2026-02-05 to 2026-07-10: no CP/CPS version published; v4.11 (2025-11-28) stays in force.

    • 2026-06-29: 2026 self-assessment submitted, 14 days into the non-compliance window, on a template already covering Chrome v1.8. The dedicated TLS hierarchy clauses are answered "CURRENTLY COMPLY and INTEND TO CONTINUE COMPLIANCE", correctly.

    • 2026-07-10: v4.12 published; the date update is applied to 7.1.2.2 instead of 7.1.2.3.

    • 2026-07-16: v4.13 published, correcting 7.1.2.3.

  • Detection: Both controls were in place before 2026-06-15 and neither found the problem. The change-triggered review had nothing to look at for most of the period, since no version was published between February and July. The self-assessment was aimed straight at the clientAuth requirement while the incident was running and still passed, because it asks whether the CP/CPS meets external requirements, not whether issuance meets the CP/CPS. The incorrect update of 2026-07-10 was not caught either, by the same four-eyes review that approved it.

  • Interaction with other factors: This let CF-1 run for five months and prevented it correcting itself in July. It follows from CF-4: there is nothing to compare the document against, and no form of the profile a tool could use.

  • Root Cause Analysis methodology used: 5-Whys

Contributing Factor #3 (CF-3): No automated control checks certificates against our own CP/CPS, and none can be derived from it in its current form

  • Description: Answering Q1 directly: no. We do not maintain custom lints, per issuing CA and per profile, that check conformance against our own CP/CPS. Pre-issuance linting is a blocking control using several well-known linters chosen to complement each other, extended by in-house RA-side code and CA-side scripts, but none of it reads the CP/CPS. Nothing automatically compares what the CP/CPS commits us to, what the systems are configured to produce, and what the certificates contain. That is why this was invisible to automation: the EKU was permitted by the CA/B Forum Requirements and by Chrome policy, so no public linter had reason to flag it.

    On Q2, we cannot produce the per-profile table, because those controls do not exist. What does exist, and we disclose it, is a set of tests over copies of the CA configuration in our Git repositories, run on change, flagging deviation from expected content written into the test by CA engineers. They are not part of our documented control framework, cover a limited number of attributes, and their coverage has not been measured. On failure they block the code from being merged, though the block can be overridden in case of false positives. They could not have caught this, because the expected content they check against is maintained by hand and this policy change was never written into it, so the tests had nothing to say about the EKU or its deadline. They also never look at issued certificates.

    On Q4, we maintain positive and negative test cases for controls tied to the BRs, ETSI and Root Store Policies, but none for CP/CPS-specific commitments. No control in production today would prevent issuance of a certificate from the affected profile carrying the EKU with a notBefore between 2026-06-15 and 2026-07-17, and none existed before 2026-06-15. Building them is an action item from this incident.

  • Timeline:

    • Throughout: no control of this kind has been in place at any point.

    • 2026-06-15 to 2026-07-15: 66,105 certificates issued; pre-issuance linting passes all of them, correctly.

  • Detection: This gap is why we did not detect the mis-issuance even though it was detectable from 2026-06-15 using the published CP/CPS and our own certificates.

  • Interaction with other factors: This removed the last line of defense and set the size of the incident, since it ran until detected by a third party. It follows from CF-4: a lint for our own commitments cannot be derived from the CP/CPS as it stands, so for now it must be written by hand.

  • Root Cause Analysis methodology used: 5-Whys

Contributing Factor #4 (CF-4): The certificate profile exists in two places with a manual process keeping them in sync: Section 7 is a product specification handled as policy text, in a format that is not directly usable by tooling (principal root cause)

  • Description: Certificate profile requirements are specified across multiple sources: Section 7 of the CP/CPS outlines these requirements in a human-readable format, the CA configuration translates them into machine-enforceable rules, and subscriber-facing documentation restates them again. Each is written by a different team, in a different format, potentially at a different time, and none is directly derived from any other, so keeping them in sync depends on a manual process. The CP/CPS copy is then binding on issuance, with revocation required within five days when a certificate does not match it, even where the certificate meets every external requirement. We maintain the same information in more than one place, with the copies reconciled only by hand. This is the problem the "don't repeat yourself" rule exists to prevent: knowledge held in more than one place is eventually changed in one and not the others.

    Three things explain why this has persisted. First, Section 7 is a product specification but is handled as policy text. A CP/CPS is a legally binding document that is written for people to read and interpret, so it is hard to see a process for it that does not involve human review in some form; Section 7 specifies field by field what the systems must produce, and the certificates issued either conform to it or they do not. We apply one drafting process, one review model and one format to both. It also explains the assumption in CF-1: relaxing an external deadline changes nothing for policy text, but it changes what a specification commits us to.

    Second, the document is large. Our current multi-purpose CP/CPS covers every certificate type across the Publicly-Trusted and eIDAS frameworks, in two languages (English, Greek), for three audiences: ourselves as the CA, Subscribers and Relying Parties. The same requirement often appears in more than one place, which is what made the incorrect update in CF-2 possible.

    Third, the profile exists only as prose. The CP/CPS is a controlled document with proper versioning and PMC approval, thus the problem is not governance. It is that text in natural language, with conditions and modal verbs spread across sections and tables, holds no structured version of the profile that a tool could test a certificate against. Producing one today means a person reading the document and writing it out again, which adds a copy rather than removing one. The Git repository tests in CF-3 are that extra copy already, and they missed the same policy update that the CP/CPS did.

    The way out is to write the profile once rather than three times: define it in a form both the systems and the checks can use, and generate the CP/CPS section, the CA configuration and the conformance checks from that one definition. A generated copy can be wrong, but it cannot drift. That is what remediation actions 6 and 8 set out to do, and the earlier items either prepare for them or provide coverage until they are in place.

  • Timeline:

    • Present since the certificate profiles were first documented in the CP/CPS and configured in the CA systems.

    • 2025-03-14: CP/CPS v4.9 is updated to incorporate the Root Store Policy update and the corresponding deadline; document and configuration stay aligned.

    • 2026-02-06: Internal teams are notified about the changes introduced in Chrome Root Store Policy v1.8 and the CA team updates the due date; no corresponding CP/CPS action is recorded.

    • 2026-06-15 00:00:01 (UTC): The TLS profiles documented in the CP/CPS require the removal of the clientAuth EKU, but the actual profiles continue to include it.

  • Detection: There is no detection event here, since this is a standing condition rather than a defect introduced on a date. It becomes visible through the incidents it produces, and it has also produced bug 2056668. It has persisted because manual coordination works nearly all the time, and because there is no established tooling for this. Linting solved the equivalent problem for the CA/B Forum Requirements; nothing plays that role for a CA's own CP/CPS, and in the incidents we reviewed the response to this class of failure was procedural in almost every case, with little public detail on implementation where automation was proposed.

  • Interaction with other factors: This is the common source of CF-1, CF-2 and CF-3. It also limits what remediation can achieve: fixing the coordination step leaves the document unverifiable, and hand-written lints add a further copy.

  • Root Cause Analysis methodology used: 5-Whys

Lessons Learned

  • What went well: The mass-revocation plan was executed correctly, and all affected certificates were revoked within the required timelines. Emergency changes remediated the cause shortly after the Certificate Problem Report was received.

  • What didn't go well: The issue was detected by a third party rather than by our own controls. We also had one opportunity to correct it ourselves, when CP/CPS v4.12 was published on 2026-07-10 to incorporate the same policy change, and the update was applied to the wrong section. After the emergency correction in v4.13, we had no defined step to review what else in the document might diverge from practice for the same reason; a second divergence of the same kind was subsequently raised by Chrome and is tracked as 2056668.

  • Where we got lucky: The divergence ran in a stricter direction. The report also reached us early in the window, since our controls did not raise it themselves.

  • Additional: Our review of comparable incidents in Bugzilla shows this is a known pain point in the ecosystem, and a fully automated control is an engineering challenge with compliance, operational and legal implications. Perhaps for this reason, remediation has usually been directed at the coordination step that failed. We have taken from that review that procedural fixes alone do not hold, and that adding hand-written conformance checks creates a further copy of the same information to keep in sync. That is why our end goal is a single definition from which the document, the configuration and the checks are generated.

Action Items

The plan is layered. Each item either creates the conditions the next depends on, or covers the gap while those conditions are built. The phases overlap in execution. In particular, the first conformance checks in Phase 3 do not wait for Phase 2 to finish, because their purpose is to give cover during the enabling work.

Comparable remediation actions at other CAs, summarized in the Related Incidents section, have consistently run longer than first projected, in one case five months and in another six and a half months beyond the original estimate. We take this to reflect the complexity of the work rather than poor planning. The dates for remediation actions 1 to 5 are commitments. Remediation actions 6 to 8 involve building something that does not yet exist, so their dates are best estimates at this stage; we will confirm or revise them at the design milestone in remediation action 8, and report either way in this bug.

Phase 1 – Stabilize and establish a baseline (actions 1–3)

Phase 1 introduces compensating controls that address the immediate issues. None of them changes the structure that produced the incident. Remediation action 1 establishes where we actually stand, which everything later depends on. Remediation actions 2 and 3 close the two gaps in how a change reaches the document.

Phase 2 – Make the specification workable (actions 4–6)

Phase 2 introduces the enablers. None of them prevents a divergence on its own and we do not present them as fixing the root cause. Automated conformance checking is not reachable without them, because a specification that is not separated from policy prose, and not in a form a tool can use, cannot feed anything. Remediation action 6 is the pivot.

Phase 3 – First conformance checks (action 7)

Phase 3 introduces a transitional control. Remediation action 7 is seeded by hand from remediation action 1 and is therefore another copy of the same knowledge, with the drift risk CF-4 describes. We accept that deliberately, on the basis that an imperfect check now is better than no check until mid-2027, and we intend it to be retired into generation under remediation action 8 rather than maintained by hand indefinitely. The date reflects that this extends existing engineering practice and CI, described in CF-3, rather than starting from nothing.

Phase 4 – Single source of truth (action 8)

Phase 4 is the target state, and the remediation action that addresses the principal root cause. Everything above either prepares for it or provides coverage until it is in production.

Consolidated table of action items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
1. CP/CPS-to-production reconciliation review. Complete the review of every normative statement on certificate content in the CP/CPS, both languages, against the profile configuration and against issued certificates. Not limited to Section 7. Disposition each difference as a document fix or a system fix, and report separately anything that is non-compliance. Output: a statement-to-profile mapping, used as the starting point for remediation actions 6 and 7. Detect CF-2, CF-3, CF-4 Number of statements reviewed and differences dispositioned, published. Verifiable through the resulting CP/CPS version history and any further incident reports. 2026-10-31 Ongoing
2. Improvement of the CP/CPS impact assessment process with recorded reasoning. Every incoming external change gets its own CP/CPS impact ticket, assessed on its own terms and not as a continuation of an earlier change, including where the conclusion is that no change is needed, with the reasoning recorded and reviewed. The assessment states what the CP/CPS currently commits to on the subject and whether that commitment is already in force, so a rollback cannot be mistaken for no action. The external change cannot be closed while the ticket is open. Detect CF-1, CF-4 Share of external changes processed after the effective date holding a CP/CPS impact ticket with recorded and reviewed reasoning. 2026-09-30 Ongoing
3. Peer compliance officer as second reviewer for normative changes. For changes arising from an incoming normative requirement, the second reviewer in the existing four-eyes review is a compliance officer with a broad view of incoming normative changes, able to establish the scope of the change independently of the drafter. That reviewer works from the originating requirement, identifies which sections it affects, and confirms each has been addressed before the change goes to the PMC. The scope determination is recorded with the change. Prevent CF-2 Share of normative CP/CPS changes after the effective date carrying a recorded scope determination by a second compliance officer. 2026-09-30 Ongoing
4. Migration of the CP/CPS to text-based markup under change control. Move the CP/CPS from .docx to text-based markup under change control, so it becomes workable by tooling and can carry enforced dependencies on other artifacts. The review process becomes more effective: per-change diffs tied to a ticket, statement-level history, and automated checks on a proposed version before approval. Publication becomes a build step. Prevent CF-4 Published CP/CPS is produced from the source repository, and the markdown URL is shared in CCADB. 2026-11-30 Ongoing
5. Separate CP/CPS documents per certificate type. Continue the work started before this incident to split the consolidated CP/CPS into separate documents per certificate type, in line with planned Apple and Chrome Root Program Policy changes. Sequenced after remediation action 4 so the split happens under change control. Mitigate CF-4 Publication of the separate documents, verifiable through our policy repository and CCADB records. 2027-03-31 Ongoing
6. Certificate profile specification separated from policy prose. Evaluate methods to separate the certificate profile specification from policy prose and implement the selected approach. Certificate content is defined in a structured, machine-readable artifact under change control, with the published CP/CPS sections generated from it. Includes moving specification statements found outside Section 7 by remediation action 1. Specification changes follow an engineering change process, so an upstream relaxation is assessed against what the specification already commits to. Prerequisite for remediation action 8. Prevent CF-4 Published profile sections are generated rather than written by hand, demonstrable from the repository. Count of specification statements moved into the specification artifact. Verifiable through the published CP/CPS. 2027-03-31 Ongoing
7. CP/CPS-specific conformance checks for TLS profiles. Automated checks of certificates against our own CP/CPS commitments for TLS profiles: pre-issuance as a blocking control, plus post-issuance as part of the self-audits process. Date-aware, so a deadline stated in the CP/CPS takes effect on the stated date without a manual step. Includes a rule for the affected profile carrying the clientAuth EKU with a notBefore in 2026-06-15 to 2026-07-17, and a separate rule for the 2027-03-15 deadline, in production well before that date. Prevent, Detect CF-3, CF-2 Coverage as [profiles covered]/[total active profiles] and [attributes covered]/[total attributes], with the production date per tranche, published. Positive and negative test cases per rule, including the two above. First tranche incl. clientAuth rules 2026-11-30; 2027-03-15 rule 2027-01-31; all active TLS profiles 2027-03-31 Ongoing
8. Single source of truth for certificate profiles. One machine-readable definition per certificate profile, built on remediation action 6, from which the CP/CPS profile sections, the CA configuration and the remediation action 7 checks are generated rather than written separately. Architecture not yet fixed; we will evaluate the approaches taken elsewhere in the ecosystem, summarized in Related Incidents, before committing. Approaches that add another hand-maintained copy without removing an existing one will not be adopted on their own. Prevent CF-4 (principal root cause), and consequently CF-2, CF-3 At the design milestone: chosen approach, coverage as [profiles covered]/[total active profiles]. At completion: one change to a profile definition demonstrably produces the corresponding change in the published document, in the CA configuration and in the checks, and the hand-maintained remediation action 7 rules are retired into generation. Design milestone 2027-01-31; TLS profiles in production 2027-06-30 Ongoing

A note on the wider gap

We agree that the answer to this class of failure is better conformance engineering rather than weaker obligations. HARICA prepared the ballot that made pre-issuance linting mandatory (SC-75), and our view here is the same. However, there is no off-the-shelf tooling for CP/CPS conformance as there is for the CA/B Forum Requirements, so remediation action 8 involves building something that does not yet exist, which is why it carries a design milestone before a production date. Where the result is reusable HARICA intends to publish it and contribute back to the community.

Appendix

Attached the population of affected certificates – including at-the-time active, expired and previously revoked – in the format required by CCADB's Incident Reporting Guidelines.

I really like the full incident report, it's showing a much better improvement than the private exchanges before.

Timeline (updated)

  • 2026-07-15 17:19: TLS Certificate profiles are updated to include the clientAuth EKU. (NON-COMPLIANCE ENDS)

  • 2026-07-15 17:21: CA Engineering team resumes issuance.

  • 2026-07-15 17:59: Mass-revocation plan is approved to begin.

  • 2026-07-16 10:45: Internal coordination meeting on mass revocation to ensure all teams are clear on their required tasks.

  • 2026-07-16 10:48: CP/CPS v4.13 is published with corrected language for the clientAuth EKU.

  • 2026-07-16 12:05: CA Engineering team enables the clientAuth EKU to the server TLS certificate profiles.

The timeline isn't entirely clear on what's happening with clientAuth EKU here and to what profiles. On first appearance it seems like it's being enabled, stopping non-compliance, then later it's being enabled again with no statement of it ever being disabled. Did you mean to have either timeline element only apply to specific profiles, and not all server TLS certificate profiles as implied?

Additionally Comment #8 still stands.

The move to a single source of truth for certificate profiles and a single set of documents per PKI use-case reflects where the ecosystem is going. I'm glad that HARICA are making an effort to get started on that, as all CA's should in the immediate future.

(In reply to Wayne from comment #8)

Even if you were requesting some extra time that isn't anything the community or root program can just approve, or apply retroactively in this case. A Full incident report MUST be provided within 14 days as per CCADB policy. You are already hours past the threshold even timing off of the start of this bug report.

Missing the 14 days MUST threshold will be another incident, which is not down to the preference of a CA. If the incidents share root causes and remediations, that will only mean it's easier for you to make the 14 day threshold in bug #2056668 in a timely manner.

If there was any awareness of the schedule being impossible prior to the deadline being broken then it should have been posted then. That would not change the result of no one being able to approve any extra time, however.

We believe the explanation provided in Comment #7 reasonably justifies the single additional day taken to submit the final elements of the report initially posted in Comment #4. This decision was made in the spirit of the Incident Reporting Guidelines and Root Store Policies. While HARICA was fully aware of the 14-day deadline, we consciously chose to spend an extra day finalizing our analysis rather than publishing a 'release candidate' that would technically meet the deadline but fall short of the expected clarity and accuracy.

We treat public incident reporting as a serious process designed to yield in-depth analysis for the benefit of the wider ecosystem. In our view, opening a separate incident solely for an explicitly justified one-day extension provides little value to the community and distracts from the substantive findings of the RCA.

That said, if the Root Program Managers determine that our approach did not meet Root Program expectations, we will readily open a new incident regarding this delay.

(In reply to Wayne from comment #10)

I really like the full incident report, it's showing a much better improvement than the private exchanges before.

Timeline (updated)

  • 2026-07-15 17:19: TLS Certificate profiles are updated to include the clientAuth EKU. (NON-COMPLIANCE ENDS)

  • 2026-07-15 17:21: CA Engineering team resumes issuance.

  • 2026-07-15 17:59: Mass-revocation plan is approved to begin.

  • 2026-07-16 10:45: Internal coordination meeting on mass revocation to ensure all teams are clear on their required tasks.

  • 2026-07-16 10:48: CP/CPS v4.13 is published with corrected language for the clientAuth EKU.

  • 2026-07-16 12:05: CA Engineering team enables the clientAuth EKU to the server TLS certificate profiles.

The timeline isn't entirely clear on what's happening with clientAuth EKU here and to what profiles. On first appearance it seems like it's being enabled, stopping non-compliance, then later it's being enabled again with no statement of it ever being disabled. Did you mean to have either timeline element only apply to specific profiles, and not all server TLS certificate profiles as implied?

The profiles were updated to remove the TLS client auth EKU. Stating 'include' instead of 'remove' was an inadvertent typographical error. Given that this entire incident revolves around the improper inclusion of the client auth EKU, the period of non-compliance logically ended with its removal.

We're monitoring this bug for any additional questions or comments.

We have no new updates to report at this time. We kindly request deferring the next update until 2026-09-18 to better align with the approaching deadlines for action items 2 and 3.

Whiteboard: [ca-compliance] [__-misissuance] → [ca-compliance] [__-misissuance] Next update 2026-09-18
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: