HARICA: Issuance of Server TLS Certificates without AIA OCSP URI against CP/CPS
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: public-incident-reports, Assigned: public-incident-reports)
Details
(Whiteboard: [ca-compliance] [ev-misissuance] [ov-misissuance] [dv-misissuance])
Attachments
(2 files)
Preliminary Incident Report
Summary
- Incident description: On 2026-07-17, the Chrome Root Program team reported a possible inconsistency in the CP/CPS language regarding AIA OCSP URI. In alignment with industry best practices, HARICA removed the AIA OCSP URI from its TLS Certificate profiles. However, we failed to update Section 7.1.2.3 of the CP/CPS to reflect that this inclusion is now optional.
All TLS Certificates issued after 2026-03-27 09:31:01 (EST) -timestamp when AIA OCSP URI was removed from the profiles- and a notBefore of 2026-07-20 22:53 (EEST) -timestamp when AIA OCSP URI was added back to the profiles- are affected and will be replaced and revoked as required by the BRs.
A full incident report will be posted no later than 2026-07-31.
- Relevant policies: Section 7.1.2.3 of the HARICA CP/CPS (before version 4.14)
- Source of incident disclosure: Third-party reported, on an existing bug https://bugzilla.mozilla.org/show_bug.cgi?id=2055551
Updated•1 month ago
|
Full Incident Report
Summary
-
CA Owner CCADB unique ID: A000035
-
Incident description: On 2026-07-17, the Chrome Root Program team reported a possible inconsistency in the CP/CPS language regarding AIA OCSP URI. In alignment with industry best practices, HARICA had removed the AIA OCSP URI from its TLS Certificate profiles. However, we failed to update Section 7.1.2.3 of the CP/CPS to reflect that this inclusion is now optional. The certificates were in compliance with the CA/B Forum Requirements, which have made the OCSP URI optional since TLS BRs v2.0.1 in 2023, but in violation of the CP/CPS. The change making it optional was prepared in a draft of a future CP/CPS version in January 2024 but was not carried into the released version in July 2024, so the requirement stood in the document for two years before the profiles changed in March 2026 and the two diverged.
-
Timeline summary:
-
Non-compliance start date: 2026-03-27
-
Non-compliance identified date: 2026-07-20
-
Non-compliance end date: 2026-07-20
-
-
Relevant policies: Section 7.1.2.3 of the HARICA CP/CPS, versions 4.8 through 4.13, which specifies the content of Subscriber TLS certificates
-
Source of incident disclosure: Third-party reported, in the bug 2055551 (comment published 2026-07-17, reviewed and confirmed on the next business day, 2026-07-20)
Impact
-
Total number of certificates: 305,642, of which 235,301 non-expired, non-revoked when the incident was confirmed.
-
Total number of "remaining valid" certificates: Zero (0). All certificates have been revoked since 2026-07-25.
-
Affected certificate types: Server TLS Certificates (DV, OV, EV)
-
Incident heuristic: The issue affects all TLS Certificates issued after 2026-03-27 09:31:01 (EET) - timestamp when AIA OCSP URI was removed from the profiles - and before 2026-07-20 22:53 (EEST) - timestamp when AIA OCSP URI was added back to the profiles. The full corpus of affected certificates is disclosed in the Appendix.
-
Was issuance stopped in response to this incident, and why or why not?: To avoid further mis-issuances, issuance was briefly stopped until the affected server TLS Certificate Profiles were updated to include the AIA OCSP URI.
-
Analysis: N/A
-
Additional considerations: The discrepancy was raised as a comment in bug 2055551 rather than as a Certificate Problem Report. We reviewed it on the next business day, confirmed the violation on 2026-07-20, and completed revocation of all affected certificates within five days of that confirmation.
Timeline
All timestamps in EEST (GMT+3) except where indicated otherwise
-
2022-10-17: HARICA CP/CPS v4.6 is released
-
2023-04-11: TLS BRs v2.0.0 are released incorporating ballot SC062 with a complete conversion of sections 7.1.2 and 7.1.4 to a tabular format
-
2023-07-17: HARICA CP/CPS v4.7 is released, without updating the formatting of sections 7.1.2 and 7.1.4 (legacy formatting of BR 1.8 is kept)
-
2023-08-17: TLS BRs v2.0.1 are released incorporating ballot SC063 making OCSP optional
-
2023-11-22: The first draft HARICA CP/CPS v5 is prepared following the structure of TLS BRs v2 with detailed certificate profiles for all certificate types
-
2024-01-16: A separate draft of HARICA's v5 CP/CPS makes OCSP optional
-
2024-07-17: HARICA CP/CPS v4.8 is released, keeping the structure of v4.7 in section 7, incorporating TLS BRs v2.0.5 but failing to backport the change that makes OCSP optional (divergence begins).
-
2025-12-05: An internal ticket is registered in the CA engineering department for the removal of OCSP URLs from TLS server certificates starting on 2026-03-02
-
2026-01-29: HARICA announces to subscribers that the OCSP URL will be removed from TLS certificates starting 2026-03-02
-
2026-03-04: Implementation of the ticket begins
-
2026-03-27 09:31 (EET): AIA OCSP URI is removed from production TLS certificates (non-compliance begins)
-
2026-06-29 15:02: HARICA completes and submits its 2026 self-assessment, based on a comparison of CP/CPS v4.11 against the CA/B Forum Requirements and Root Store Policies. The divergence subject of this report is not detected.
-
2026-07-17: The Chrome Root Program team publishes a comment in bug 2055551 raising a possible inconsistency between the CP/CPS language on the AIA OCSP URI and HARICA's issued certificates.
-
2026-07-20 21:30: The comment is reviewed on the next business day. A compliance meeting is held with department leads; the violation is confirmed.
-
2026-07-20 22:30: Certificate issuance is halted
-
2026-07-20 22:53: Affected certificate profiles are updated to include the AIA OCSP URI (non-compliance ends)
-
2026-07-20 23:00: Certificate issuance is resumed
-
2026-07-21: Confined CP/CPS review and update to make the AIA OCSP URI optional for TLS certificates
-
2026-07-21 15:54: Notification of affected Subscribers is initiated
-
2026-07-21 18:49: CP/CPS v4.14 published
-
2026-07-21 19:09: The AIA OCSP URI is removed again from the TLS certificate profiles, now permitted by CP/CPS v4.14
-
2026-07-21 19:45: Legacy ARI configured for re-issuance
-
2026-07-21 21:56: Preliminary incident report is published in bug 2056668 (this bug)
-
2026-07-22 11:48: Flexible ARI configured for re-issuance
-
2026-07-25 13:49: Affected certificate SANs replacement rate at 87%
-
2026-07-25 14:00: Revocation of remaining valid affected certificates begins
-
2026-07-25 14:08: Revocation of affected certificates completed
-
2026-07-25 to 2026-07-31: Collection of the events that led to this incident, Root Cause Analysis, Lessons Learned and Remediation Action Plan in correlation with bug 2055551, compilation of the Draft Incident Report
Related Incidents
For related incidents associated with this bug, please refer to the Final Incident Report in bug 2055551, section "Related Incidents".
One of them matches this incident almost exactly. In bug 1962830, Microsoft PKI Services removed the OCSP URI from TLS certificates before publishing the CP/CPS update that made it optional, so the practice moved ahead of the document on the same subject and in the same direction as here. That incident was also reported by a third party, and its remediation was procedural: the project plan template was updated to include a CP/CPS conflict check prior to CA changes, without tooling or automation. Our remediation action 9 addresses the same step in our own process, and we have deliberately not stopped there, for the reasons set out in the Action Items of bug 2055551.
The reason we have not stopped there is visible in the same set of bugs. Four months after 1962830, the same CA reported 1979475, a further divergence between its CP/CPS and its issued certificates, and that report identified the misalignment between linting tools and CPS enforcement as a contributing factor. Our own pair of incidents follows the same shape: bug 2055551 and this one were raised days apart, from the same root cause and in the same document. In both cases a procedural fix to the step that failed did not prevent the next occurrence, because the two copies of the certificate profile were still kept in step by hand.
Root Cause Analysis
We worked back from the issuance using a 5-Whys chain, in correlation with the analysis of bug 2055551.
In short: the CP/CPS required an AIA OCSP URI in Subscriber TLS certificates because a change making it optional was prepared in a draft of CP/CPS v5 in January 2024 and was not carried into the released v4.8 in July 2024. Nothing since compared the document against the certificate profiles, so the requirement stood for two years. When the AIA OCSP URI was removed from production profiles in March 2026, the change was made on the understanding that the CP/CPS already permitted it, and the document and the profiles diverged.
Based on this analysis and the RCA of bug 2055551, the two incidents share most of the Contributing Factors, with a few adjustments and one addition. With the exception of CF-1 ("A Root Program relaxation was treated as needing no CP/CPS action") of bug 2055551, all the other CFs (CF-2 to CF-4) are also applicable to this bug.
This mapping is presented here:
| Contributing factors of bug 2055551 | Applicability in bug 2056668 (this bug) | |
|---|---|---|
| CF-1 | A Root Program relaxation was treated as needing no CP/CPS action | Not applicable |
| CF-2 | CP/CPS reviews check the document only against external requirements, and did not catch the v4.12 update landing in the wrong section | The first part is directly applicable in this case too, as we are dealing with another case of a CP/CPS unintentionally setting a stricter policy without the practice following through. The second part of the title refers to the CP/CPS v4.12 update in bug 2055551 and does not apply here. The 2026 self-assessment, submitted on 2026-06-29 while this divergence was live, did not surface it for the same reason. |
| CF-3 | No automated control checks certificates against our own CP/CPS, and none can be derived from it in its current form | This factor is directly applicable in this case too. It is further enhanced, due to the legacy non-tabular format of Section 7 of the HARICA CP/CPS, which makes the comparison and transposition of external updates to certificate profiles error prone. |
| CF-4 | The certificate profile exists in two places with a manual process keeping them in sync: Section 7 is a product specification handled as policy text, in a format that is not directly usable by tooling (principal root cause) | This factor is directly applicable in this case too, and it is the principal root cause here as well: the two copies were only ever reconciled by hand. |
The analysis of the events that resulted in this violation revealed the following additional Contributing Factor:
Contributing Factor #5 (CF-5): A CP/CPS change prepared in the v5 draft was not carried into the released v4.x line, and no step verified it
-
Description: HARICA was in the process of a major editorial work to update the CP/CPS with a detailed section 7 that matches the tabular format introduced in TLS BRs v2.0.0. During a re-structuring of this kind the available comparison tools have limited value, because everything shows as additions and deletions rather than as changes to the content, and that risk has been a longstanding deterrent for the completion of the revamp: Section 7 still follows the legacy format more than three years after the TLS BRs moved to the tabular one. The update to section 7.1.2.3 was applied to the draft revamp, intended for CP/CPS version 5.0, and when we decided to postpone the adoption of version 5.0 the revert process failed to backport it to the version 4.x line. All teams considered that the CP/CPS change to allow the removal of the AIA OCSP URI was completed; none verified. When HARICA decided to remove it from the end-entity certificate profile in March 2026, there was no pending ticket or work related to a CP/CPS change and the CA Engineering team proceeded with the actual certificate profile change.
We consider this a failure of the Change Management process. As a remediation action, before making any changes to a CA or end-entity certificate profile, we will add an explicit step to our change management workflow to double-check the CP/CPS for potential incompatibilities. Approval will require a review by both a Compliance Officer and a CA Engineer, as well as documentation of the specific CP/CPS clauses examined.
The same failure could have affected other changes prepared in the v5 draft and not carried across. Remediation action 1 of bug 2055551 reviews every normative statement on certificate content in the CP/CPS, in both languages, against the profile configuration and against issued certificates. That is the check that would surface any of them, and it is why we have not opened a separate review limited to the v5 backport.
-
Timeline:
-
2023-08-17: TLS BRs v2.0.1 are released incorporating ballot SC063 making OCSP optional
-
2023-11-22: The first draft HARICA CP/CPS v5 is prepared following the structure of TLS BRs v2
-
2024-01-16: A separate draft of HARICA's v5 CP/CPS makes OCSP optional
-
2024-07-17: HARICA CP/CPS v4.8 is released, keeping the structure of v4.7 in section 7 and failing to backport the change that makes OCSP optional
-
2026-03-27 09:31 (EET): AIA OCSP URI is removed from production TLS certificates, on the understanding that the CP/CPS already permitted it
-
2026-07-20 21:30: The violation is confirmed, two years after the divergence was introduced
-
-
Detection: Not detected internally at any point in the two years between the divergence being introduced in v4.8 and the report. No step in the CP/CPS release process compares the published version against the draft it was derived from, so a change that was made in the draft and not carried across leaves no trace. The profile change of March 2026 was made against a CP/CPS that all teams considered already permitted it, so nothing in that change prompted a re-reading of Section 7.1.2.3. The divergence was raised by the Chrome Root Program team on 2026-07-17, while reviewing the same section in the course of bug 2055551.
-
Interaction with other factors: This is what caused the two copies described in CF-4 to diverge in this case, as CF-1 did in bug 2055551. CF-2 and CF-3 explain why the divergence stood for two years. This factor is addressed by the change management step in remediation action 9, and anything else lost in the same backport is covered by remediation action 1 of bug 2055551; the two plans are complementary and we treat them as one.
-
Root Cause Analysis methodology used: 5-Whys
Lessons Learned
-
What went well: HARICA's mass-revocation plan was executed without issues. Once the violation was confirmed, issuance was halted and the certificate profiles corrected within 83 minutes, and the preliminary incident report was published the following day. 87% of the affected SANs had been issued replacement certificates before revocation began.
-
What didn't go well: An avoidable error caused the mass replacement and revocation of TLS Certificates. The divergence stood for two years and was found by the Chrome Root Program team rather than by us. Our first response to bug 2055551 also corrected the clause that had been reported without asking what else in Section 7 might diverge from practice for the same reason.
-
Where we got lucky: The divergence ran in a stricter direction, and the practice was the correct one: removing the AIA OCSP URI follows the industry's move away from OCSP. The report also reached us while a related section of the same document was already under external scrutiny; nothing in our own controls was going to raise it.
-
Additional: This incident and bug 2055551 share a principal root cause and most of their contributing factors, which is why the remediation plan is common to both. The difference between them is instructive: in bug 2055551 the document was left behind by an external change, here the practice moved ahead of the document. Controls that look only in one direction cannot catch both.
Action Items
The remediation plan for this incident includes all the action items of bug 2055551 (which we do not repeat here), plus the following action item:
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| 9. Update of the Change Management Procedure with regards to Certificate Profiles. Before any change to a CA or end-entity certificate profile, the change management workflow requires an explicit check of the CP/CPS for incompatibilities. Approval requires review by both a Compliance Officer and a CA Engineer, and the specific CP/CPS clauses examined are recorded with the change. | Prevent, Detect | CF-5 | Share of certificate profile changes after the effective date carrying a recorded CP/CPS compatibility check, signed off by a Compliance Officer and a CA Engineer, listing the clauses examined. | 2026-08-14 | Ongoing |
Appendix
Attached [1], [2] the population of affected certificates – including at-the-time active, expired and previously revoked – in the format required by CCADB's Incident Reporting Guidelines.
Noting some issues.
Impact
Total number of certificates: 305,642, of which 235,301 non-expired, non-revoked when the incident was confirmed.
Total number of "remaining valid" certificates: Zero (0). All certificates have been revoked since 2026-07-25.
Affected certificate types: Server TLS Certificates (DV, OV, EV)
Incident heuristic: The issue affects all TLS Certificates issued after 2026-03-27 09:31:01 (EET) - timestamp when AIA OCSP URI was removed from the profiles - and before 2026-07-20 22:53 (EEST) - timestamp when AIA OCSP URI was added back to the profiles. The full corpus of affected certificates is disclosed in the Appendix.
Your incident heuristic is not correct. There is evidence of two periods of time when profiles were active with AIA OCSP URI during this timeframe:
- 2026-05-07T07:00:14Z to 2026-05-07T07:20:07Z (not on crt.sh, but is on Censys)
- 2026-05-12T06:52:26Z to 2026-05-12T10:10:07Z.
These batches total 334 certificates in total, 20 on 2026-05-07 and 310 on 2026-05-12.
Given these are unrevoked it should have came up during your review. The reference in Root Cause Analysis to AIA OCSP URI being removed in 'production profiles' also makes it clear that we were supposed to be presented with a full and complete picture of profile changes. That is not the case, and I recommend having an inventory of all 'production profiles' to ensure you have not missed any.
Related Incidents
For related incidents associated with this bug, please refer to the Final Incident Report in bug 2055551, section "Related Incidents".
For 'Related Incidents' and other aspects of this full incident report: you cannot refer to another incident. These are supposed to be self-contained reports. This is a very messy approach that will cause confusion as you attempt to explain action items in different incidents. While they may share the same root causes that does not mean you should mix and match incident reports.
All reports MUST be free-standing and not rely on the contents of other reports. While reports MAY repeat information from discussions or Bugzilla comments, they SHOULD summarize previous findings. CA Owners are responsible for compiling a complete report according to these guidelines, even if information exists elsewhere.
(In reply to Wayne from comment #4)
Noting some issues.
Impact
Total number of certificates: 305,642, of which 235,301 non-expired, non-revoked when the incident was confirmed.
Total number of "remaining valid" certificates: Zero (0). All certificates have been revoked since 2026-07-25.
Affected certificate types: Server TLS Certificates (DV, OV, EV)
Incident heuristic: The issue affects all TLS Certificates issued after 2026-03-27 09:31:01 (EET) - timestamp when AIA OCSP URI was removed from the profiles - and before 2026-07-20 22:53 (EEST) - timestamp when AIA OCSP URI was added back to the profiles. The full corpus of affected certificates is disclosed in the Appendix.
Your incident heuristic is not correct. There is evidence of two periods of time when profiles were active with AIA OCSP URI during this timeframe:
- 2026-05-07T07:00:14Z to 2026-05-07T07:20:07Z (not on crt.sh, but is on Censys)
- 2026-05-12T06:52:26Z to 2026-05-12T10:10:07Z.
These batches total 334 certificates in total, 20 on 2026-05-07 and 310 on 2026-05-12.
Given these are unrevoked it should have came up during your review. The reference in Root Cause Analysis to AIA OCSP URI being removed in 'production profiles' also makes it clear that we were supposed to be presented with a full and complete picture of profile changes. That is not the case, and I recommend having an inventory of all 'production profiles' to ensure you have not missed any.
Although the Incident Reporting Guidelines permit omitting the 'incident heuristic' field when a complete list of affected certificates is provided in the appendix, we chose to include a simple heuristic to aid in high-level impact assessment. By nature, a heuristic is an approximation. During this period, we encountered Subscribers with strict, unanticipated operational dependencies on OCSP. When no alternative workarounds existed, we selectively permitted the inclusion of the OCSP URI in certificates issued from specific CAs during pre-agreed time windows. Because these specific certificates successfully included the OCSP locator, they were correctly excluded from the list of affected certificates.
Related Incidents
For related incidents associated with this bug, please refer to the Final Incident Report in bug 2055551, section "Related Incidents".
For 'Related Incidents' and other aspects of this full incident report: you cannot refer to another incident. These are supposed to be self-contained reports. This is a very messy approach that will cause confusion as you attempt to explain action items in different incidents. While they may share the same root causes that does not mean you should mix and match incident reports.
All reports MUST be free-standing and not rely on the contents of other reports. While reports MAY repeat information from discussions or Bugzilla comments, they SHOULD summarize previous findings. CA Owners are responsible for compiling a complete report according to these guidelines, even if information exists elsewhere.
Given the extensive detail already provided in Bug 2055551, which shares many similarities with this incident, we opted to reference it to avoid unnecessary duplication. As noted in the Root Cause Analysis section, we believe the contributing factors summary table provides sufficient context for readers to fully understand this incident. However, if the community finds this referenced format insufficient, we are happy to provide a revised version that directly incorporates the relevant contents of Bug 2055551.
We're also monitoring this bug for any additional questions or comments.
This is an update on the action item which is unique to this bug:
| Action Item | Kind | Corresponding Root Cause(s) | Evaluation Criteria | Due Date | Status |
|---|---|---|---|---|---|
| 9. Update of the Change Management Procedure with regards to Certificate Profiles. Before any change to a CA or end-entity certificate profile, the change management workflow requires an explicit check of the CP/CPS for incompatibilities. Approval requires review by both a Compliance Officer and a CA Engineer, and the specific CP/CPS clauses examined are recorded with the change. | Prevent, Detect | CF-5 | Share of certificate profile changes after the effective date carrying a recorded CP/CPS compatibility check, signed off by a Compliance Officer and a CA Engineer, listing the clauses examined; audited against the change register. Limited public measurability, stated plainly. | 2026-08-14 | Completed (2026-08-13) |
All remaining action items are now tracked in bug 2055551. We propose submitting a closure summary for this bug and providing future updates exclusively in bug 2055551. Please let us know if you have any objections.
Report Closure Summary
-
Incident description:
HARICA issued 305,642 Server TLS Certificates (DV, OV, and EV) between 2026-03-27 and 2026-07-20 without an AIA OCSP URI, contrary to Section 7.1.2.3 of HARICA's CP/CPS versions 4.8 through 4.13. Although the certificates complied with the applicable CA/Browser Forum TLS Baseline Requirements, which made the inclusion of an OCSP URI optional, they did not comply with HARICA's own CP/CPS, which continued to require the URI. The discrepancy was identified following a report by the Chrome Root Program team. -
Incident Root Cause(s):
The principal root cause was the maintenance of certificate-profile requirements in two separate representations: the CP/CPS and the certificate-profile configuration. These representations were reconciled manually, without an automated mechanism to detect inconsistencies between the published CP/CPS and production certificate profiles.A change making the AIA OCSP URI optional had been prepared in a draft of CP/CPS version 5 in January 2024, but the change was not carried into the released version 4.x CP/CPS. Consequently, the CP/CPS continued to require the OCSP URI for approximately two years. When HARICA removed the OCSP URI from the production TLS certificate profiles in March 2026, the change was made under the assumption that the CP/CPS already permitted the practice.
Contributing factors included the fact that CP/CPS reviews primarily checked the document against external requirements rather than against the actual certificate profiles, the absence of an automated control comparing issued certificates with HARICA's CP/CPS, and the legacy non-tabular format of Section 7 of the CP/CPS, which made manual comparison and synchronization error-prone. The 2026 self-assessment also did not detect the divergence.
-
Remediation description:
Upon confirmation of the incident on 2026-07-20, HARICA immediately halted certificate issuance and restored the AIA OCSP URI to the affected TLS certificate profiles. Issuance resumed after the profiles were corrected. HARICA subsequently updated CP/CPS version 4.14 to explicitly make the AIA OCSP URI optional for TLS certificates, after which the certificate profiles were updated accordingly.HARICA notified affected subscribers and configured both legacy and flexible ARI mechanisms to facilitate certificate replacement. Replacement certificates were issued where applicable, and revocation of the remaining affected certificates began on 2026-07-25 and was completed eight minutes later. As a result, there are no remaining valid affected certificates.
HARICA also updated its Change Management Procedure for CA and end-entity certificate profiles. Effective 2026-08-13, changes to certificate profiles require an explicit CP/CPS compatibility check, review and approval by both a Compliance Officer and a CA Engineer, and documentation of the specific CP/CPS clauses examined.
-
Commitment summary:
HARICA is committed to strengthening the controls used to maintain consistency between its CP/CPS and certificate-profile configurations. The updated change-management process provides a formal control requiring CP/CPS compatibility verification before certificate-profile changes are approved.In addition, HARICA is addressing the underlying systemic weakness identified during the incident: certificate-profile requirements were maintained in separate representations and kept synchronized manually. HARICA will continue to improve its ability to compare normative CP/CPS requirements against certificate-profile configurations and issued certificates, with the goal of reducing reliance on manual synchronization and enabling earlier detection of discrepancies.
HARICA will monitor the effectiveness of these controls through its ongoing compliance and change-management processes.
All Action Items disclosed in this report have been completed as described, and we request its closure.
Comment 10•15 days ago
|
||
This is a final call for comments or questions on this Incident Report.
Otherwise, it will be closed on approximately 2026-08-31.
Updated•8 days ago
|
Description
•