2071491 - SSL.com: MPIC not performed for specific DCV path

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[SSL.com]

Preliminary Incident Report

Summary

• Incident description:
  During our annual WebTrust audit, it was discovered that we did not have MPIC corroboration evidence on the DCV performed for 4 TLS certificates amongst those sampled. A full investigation was conducted, and ultimately found a total of 2,700 TLS unexpired, unrevoked certificates where this occurred.
  We invoked our Mass Revocation Plan and successfully revoked all 2,700 TLS certificates less than 24 hours after discovering the incident.

• Relevant policies:

TLS Baseline Requirements, v2.3.0, section 3.2.2.9, SSL.com CP/CPS v1.33, section 3.2.2.13
Multi-Perspective Issuance Corroboration Multi-Perspective Issuance Corroboration attempts to corroborate the determinations (i.e., domain validation pass/fail, CAA permission/prohibition) made by the Primary Network Perspective from multiple remote Network Perspectives before Certificate issuance. This process can improve protection against equally-specific prefix Border Gateway Protocol (BGP) attacks or hijacks...

• Source of incident disclosure: Audit.

We will post the full incident report on or before 2026-09-25.

Comment 1

[SSL.com]

Attachment: mpic_incident_annex.csv

Comment 2

[SSL.com]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A002038

• Incident description:

During our annual WebTrust audit, it was discovered that we did not have MPIC corroboration evidence on the DCV performed for 4 TLS certificates amongst those sampled. A full investigation was conducted, and ultimately found a total of 2,700 TLS unexpired, unrevoked certificates where this occurred. An alternate DCV path existed in our RA platform, one of 3 issuance platforms that we currently have, that had not been attached to the MPIC implementation. The existing monitoring was on the MPIC side, not the DCV side, which meant that our monitoring did not see that the MPIC requests for these certificates were not being sent.

We invoked our Mass Revocation Plan and all 2,700 TLS certificates were revoked or expired in less than 24 hours after discovering the incident.

• Timeline summary:

  • Non-compliance start date: 2025-03-15 – When the requirement to perform MPIC was first in effect from the TLS BRs.

  • Non-compliance identified date: 2026-09-09 22:35 – Investigation into follow up on MPIC evidence from WebTrust auditors discovered the MPIC evidence was not present.

  • Non-compliance end date: 2026-09-10 20:25 UTC – Mass revocation event is completed for all unrevoked, unexpired certificates.

• Relevant policies:

TLS Baseline Requirements, v2.3.0, section 3.2.2.9, SSL.com CP/CPS v1.33, section 3.2.2.13

Multi-Perspective Issuance Corroboration Multi-Perspective Issuance Corroboration attempts to corroborate the determinations (i.e., domain validation pass/fail, CAA permission/prohibition) made by the Primary Network Perspective from multiple remote Network Perspectives before Certificate issuance. This process can improve protection against equally-specific prefix Border Gateway Protocol (BGP) attacks or hijacks...

• Source of incident disclosure: Audit.

Impact

• Total number of certificates: 6,031 total TLS certificates, including expired and previously revoked.

• Total number of "remaining valid" certificates: 2,700 non-expired, non-revoked certificates, all of which are now revoked or expired.

• Affected certificate types: For the unexpired, unrevoked certs: 2,418 DV TLS, 280 OV TLS, and 2 EV TLS.

• Incident heuristic: The full corpus of affected certificates is attached.

• Was issuance stopped in response to this incident, and why or why not?: Yes, new validation TLS issuance was stopped from our RA platform. See timeline and Contributing Factor #4 for more discussion on this item.

• Analysis: N/A.

• Additional considerations: This incident affected our RA platform, which is entirely separate from the EJBCA system that was affected in Bug 2029230.

Timeline

All times are UTC unless where otherwise noted.

2018–2020 — The RA platform's DCV logic is built using two DCV paths, a primary and a secondary path, called in different circumstances, both satisfy a DCV request identically. No inventory of DCV entry points is maintained. Only one of the paths are utilized for DCV operations.

2018 - A domain pre-validation feature is built inside the RA Platform using the secondary DCV path, out of the normal DCV flow and path.

2025-02 - MPIC is implemented in our RA platform. Enforcement is placed at the primary DCV path in which that method calls the MPIC service before validation and halts issuance if corroboration fails. The secondary path is not modified and remains un-gated. MPIC results are written to audit log tables and are not persisted on the DCV record.

2025-03-15 - TLS BR Section 3.2.2.9 MPIC requirements become mandatory. From this date, any DCV completed through the secondary path is non-conforming.

2025-08-10 - Earliest certificate in the unexpired, unrevoked population is issued.

2026-09-09 19:32 - SSL.com's auditors communicate a follow-up regarding MPIC evidence for 16 certificate samples.

2026-09-09 19:35 - Investigation begins for the follow up.

2026-09-09 21:27 – 12 of the 16 follow-ups are verified, 4 are under further investigation.

2026-09-09 22:35 – Investigation confirms that MPIC was not performed for 4 TLS certificates from the audit samples.

2026-09-10 02:02 - Preliminary root cause identified: MPIC is gated on the primary DCV method only; the secondary method is un-gated.

2026-09-10 02:04 – MPIC enforcement moved inside the DCV verification method itself, closing the bypass for present and future callers.

2026-09-10 03:14 - Patch deployed to production, which stops all new issuances through our RA system that required new validations, but unintentionally does not block reuse of existing DCV that did not perform MPIC.

2026-09-10 04:17 - Non-TLS issuance resumed; TLS issuance for new validations remains blocked.

2026-09-10 07:44 - The bypass mechanism is confirmed against production data. Read-only queries begin, enumerating affected DCVs and the certificates issued in reliance on them.

2026-09-10 09:32 - Raw MPIC logs collected from all four validation servers, completing the evidence set.

2026-09-10 07:00-16:00 - Investigation and population building continue.

2026-09-10 15:06 - Incident coordination channel established across Engineering, Compliance, Validation, Support, Marketing and Leadership.

2026-09-10 16:24 - Preliminary population communicated, indicating certificate population to be over 2,000 certificates.

2026-09-10 16:34 - Mass Revocation Plan is confirmed necessary due to current count of certificates involved, and Bulk Revocation Procedure retrieved.

2026-09-10 17:20-17:54 - Invalidation of all affected reusable DCVs completed. Certificate requests against those domains will go through new DCV and MPIC before issuance.

2026-09-10 18:05 - Confirmation that all affected certificates have been identified: 2,700 certificates.

2026-09-10 18:15 - Internal record opened for Mass Revocation Plan execution; the revocation list is attached to it.

2026-09-10 18:18 - Senior leadership approval to commence the Mass Revocation Plan.

2026-09-10 18:29 - TLS issuance re-enabled.

2026-09-10 19:02 - Bulk revocation commences. Revocations take effect immediately via OCSP.

2026-09-10 20:25 - Bulk Revocation Procedure run completes with 2,697 total certificates revoked, and 3 expired before the completion of the bulk revocation.

2026-09-10 20:39 - Updated CRLs posted.

2026-09-11 21:27 - Preliminary Incident Report filed.

2026-09-25 21:17 – Full incident report filed.

Related Incidents

Bug	Date	Description
Bug 2029230	2026-04-03	SSL.com MPIC incident, where a defect in a third-party component on the ACME path resulted in only remote perspectives being performed.
Bug 2029643	2026-04-06	HARICA MPIC incident, where a defect in a third-party component on the ACME path resulted in only remote perspectives being performed.

Root Cause Analysis

MPIC was implemented as a call-site feature rather than as an issuance-time invariant, in a codebase with duplicated and unmapped DCV entry points, and no independent detective control reconciled issued certificates against MPIC evidence.

Contributing Factor #1: Duplicated DCV entry points with no authoritative inventory

• Description: The RA platform performs DCV through two distinct entry points - a primary and secondary method - that satisfy a validation identically and write indistinguishable validation records. The duplication is a legacy artifact of the 2018–2020 codebase. No inventory or map of DCV code paths was maintained, and no architectural rule required compliance-critical operations to flow through a single choke point. This made "implement MPIC" a discovery problem for which no map existed: an implementer could only gate the paths they knew about.

• Timeline: The duplication was introduced in 2018–2020, before MPIC existed. It became a compliance defect on 2025-03-15, when TLS BR Section 3.2.2.9 took effect with one of the two paths un-gated.

• Detection: Detected during evidence collection for sample selections by SSL.com's auditors.

• Interaction with other factors: This factor is the substrate for all the others. Contributing Factor #2 is what left the duplicate path un-gated; Contributing Factor #3 is the reason the result went undetected for eighteen months.

• Root Cause Analysis methodology used: 5 Whys.

Contributing Factor #2: MPIC enforcement placed in the callers rather than in the callee

• Description: When MPIC was implemented in February 2025, the gate was added at the call sites of the primary DCV path - the primary method invokes the MPIC service before validation and halts issuance on a corroboration failure - rather than inside the method that actually performs verification. Enforcement placed in callers silently exempts every caller not known to the implementer at the time. The secondary method continued to perform validation through a single-vantage blocking driver, with no MPIC call in its execution path.

• Timeline: Introduced February 2025 with the MPIC implementation. Became non-conforming on 2025-03-15. Corrected on 2026-09-10 at 02:04 UTC, when enforcement was moved inside the verification methods.

• Detection: Not detected. The implementation was reviewed, tested and signed off, and every one of those activities confirmed that MPIC was present where it had been implemented. It did not occur to test whether MPIC evidence existed for every issuance, instead.

• Interaction with other factors: Combined with Contributing Factor #1 to produce the defect: duplication created a second path, and call-site enforcement failed to cover it. Combined with Contributing Factor #3 to make the result invisible, since a non-MPIC validation is indistinguishable from a compliant one in the DCV record.

• Root Cause Analysis methodology used: 5 Whys.

Contributing Factor #3: No issuance-time check, and MPIC evidence not placed where it could be checked.

• Description: The final pre-issuance check did not assert that MPIC evidence exists for the DCV relied upon for every SAN. A certificate was therefore structurally issuable without it. Compounding this, MPIC results persisted only in audit log tables and were not written onto the DCV record itself, which meant that the fact needed for an issuance-time assertion was not available where issuance-time code could cheaply read it. The absence of the check and the inaccessibility of the evidence reinforced each other.

• Timeline: Present from the February 2025 MPIC implementation onward. See Action Item #2 for remediation.

• Detection: This is the absence of a control where the need for it was not detected.

• Interaction with other factors: This is what allowed Contributing Factors #1 and #2 to result in actual misissuance rather than a latent code defect.

• Root Cause Analysis methodology used: 5 Whys.

Contributing Factor #4: Sufficient incident response tooling did not exist in our RA platform before the incident.

• Description: There was no configuration-driven kill-switch for issuance: both the full stop and the TLS-only block had to be implemented by ad-hoc hot patches and a release cut, under time pressure, and the TLS-only block was not effective, with the result that 9 certificates were issued during part of the window in which TLS issuance was intended to be blocked. These certificates had existing validations in place.

• Timeline: Pre-existing. Surfaced between 2026-09-10 03:14 UTC and 17:54 UTC.

• Detection: Detected during incident response.

• Interaction with other factors: Did not cause the misissuance, but affected its size and the integrity of containment, which resulted in an additional 9 misissuances.

• Root Cause Analysis methodology used: 5 Whys.

Lessons Learned

• What went well:

  • Time from finding the incident to completed Mass Revocation of 2,700 certificates was completed within the required 24 hour timeline.

  • Prior incident experience - specifically Bug 2029230 in April 2026 - materially accelerated correct decision-making on the revocation clock and the Mass Revocation Plan. Institutional incident memory had direct operational value here.

• What didn’t go well:

  • MPIC was implemented as a feature at call sites rather than as inside the verification routine, so every caller unknown to the implementer was silently exempt.

  • Two duplicated DCV entry points existed with no inventory or map.

  • There was no issuance-time assertion that MPIC evidence exists for every SAN, so a non-compliant certificate was structurally issuable.

  • MPIC results were placed only in audit log tables and not on the DCV record, which made automated verification at issuance time impractical and post-hoc verification more difficult.

• Where we got lucky:

  • The unexpired, unrevoked affected population was 2,700 certificates rather than much higher. That is a consequence of which flows happened to call the un-gated method and how frequently they were used, not of any control we had in place. The same defect on a more heavily used path would have produced a far larger number of affected certificates over the same time period.

  • The incident was found during a scheduled audit rather than by an adversary or a researcher demonstrating a practical BGP-based attack against a domain.

• Additional: N/A.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
1. Move MPIC enforcement inside the shared DCV verification method (the callee), so that the gate applies to every caller present and future, and cover the domains-tab caller explicitly.	Prevent	CF #1, CF #2	No DCV can complete without an MPIC call in its execution path; demonstrated by a negative test in which a direct invocation of each entry point is rejected absent corroboration.	2026-09-10	Complete
2. Add an issuance-time assertion in the final pre-issuance check: reject issuance for any SAN whose freshest satisfied DCV lacks MPIC evidence. Persist MPIC evidence on the DCV record so the assertion can be made without querying audit log tables.	Prevent	CF #3	A certificate cannot be issued without recorded MPIC evidence; demonstrated by a negative test that attempts issuance against a synthetic non-MPIC DCV and is refused.	2026-10-31	Ongoing
3. Consolidate DCV to a single entry point and deprecate the secondary method, removing the un-gated method from the codebase.	Prevent	CF #1	The secondary method entry point no longer exists; verified by static analysis showing a single DCV verification callee and by the absence of any alternative path in the call graph.	2026-10-31	Ongoing
4. Deploy a recurring automated reconciliation comparing every issued certificate against stored MPIC evidence, with alerting on any certificate whose relied-upon DCV lacks corroboration evidence.	Detect	CF #3	The job runs on a defined schedule and alerts.	2026-11-15	Ongoing
5. Establish and maintain an authoritative map of all code paths that perform or consume DCV, and make its currency a precondition for approving any change to validation logic.	Prevent	CF #1	The map exists, is version-controlled, and is referenced in the change-management record of every validation-related change.	2026-11-15	Ongoing
6. Implement configuration-driven issuance kill-switches per certificate type and per issuance channel, removing the need for code changes or hot patches during containment, and make each switch's effectiveness continuously verifiable.	Mitigate	CF #4	A switch can be engaged and verified without a code deployment.	2026-11-30	Ongoing
7. Expand the automated compliance test suite to assert the MPIC invariant across every DCV entry point and every validation method permitted by the CP/CPS, including explicit negative tests for each.	Prevent, Detect	CF #1, CF #3	Test coverage report shows a positive and a negative MPIC test for each DCV entry point and validation method.	2026-11-30	Ongoing

Appendix

mpic_incident_annex.csv

The full corpus of the 2,700 affected certificates, including the 3 that expired prior to the Mass Revocation Event starting and therefore were not revoked, is provided as a comma-separated listing with the fields required by the CCADB Incident Reporting Guidelines.

Comment 3

[SSL.com]

We are actively progressing through the remaining Action Items identified in our full incident report. Action Items #2 and #3 remain ongoing and are on track for their 2026-10-31 due date. Action Items #4 through #7 remain ongoing and are on track for their 2026-11-15 and 2026-11-30 due dates, respectively.

SSL.com continues to monitor this bug for any comments or questions. We will provide an update on or before 2026-10-09.

Comment 4

[SSL.com]

We are still actively progressing through the Action Items identified in our full incident report. Action Items #2 and #3 are still on track for their 2026-10-31 due date. Action Items #4 through #7 are still on track for their 2026-11-15 and 2026-11-30 due dates, respectively.

SSL.com continues to monitor this bug for any comments or questions. We request the next update for this to be set for 2026-10-30.

Comment 5

[SSL.com]

Work on the remaining Action Items identified in our full incident report continues to progress. Action Items #2 and #3 remain on track for their 2026-10-31 due date. Action Items #4 and #5 remain on track for their 2026-11-15 due date, and Action Items #6 and #7 remain on track for their 2026-11-30 due date.

SSL.com continues to monitor this bug for any comments or questions. As requested in our previous update, we ask that the next update for this bug be set to 2026-10-30.

Comment 6

[SSL.com]

This is an update to report the completion of our remaining Action Items.

Action Items #2, #3, #6, and #7 identified in our Full Incident Report have been completed. The table below lists each Action Item with its completion date, its status, and how it was verified against its Evaluation Criteria.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
2. Add an issuance-time assertion in the final pre-issuance check: reject issuance for any SAN whose freshest satisfied DCV lacks MPIC evidence. Persist MPIC evidence on the DCV record so the assertion can be made without querying audit log tables.	Prevent	CF #3	A certificate cannot be issued without recorded MPIC evidence; demonstrated by a negative test that attempts issuance against a synthetic non-MPIC DCV and is refused.	2026-10-31	Completed
3. Consolidate DCV to a single entry point and deprecate the secondary method, removing the un-gated method from the codebase.	Prevent	CF #1	The secondary method entry point no longer exists; verified by static analysis showing a single DCV verification callee and by the absence of any alternative path in the call graph.	2026-10-31	Completed
4. Deploy a recurring automated reconciliation comparing every issued certificate against stored MPIC evidence, with alerting on any certificate whose relied-upon DCV lacks corroboration evidence.	Detect	CF #3	The job runs on a defined schedule and alerts.	2026-11-15	Ongoing
5. Establish and maintain an authoritative map of all code paths that perform or consume DCV, and make its currency a precondition for approving any change to validation logic.	Prevent	CF #1	The map exists, is version-controlled, and is referenced in the change-management record of every validation-related change.	2026-11-15	Ongoing
6. Implement configuration-driven issuance kill-switches per certificate type and per issuance channel, removing the need for code changes or hot patches during containment, and make each switch's effectiveness continuously verifiable.	Mitigate	CF #4	A switch can be engaged and verified without a code deployment.	2026-11-30	Completed
7. Expand the automated compliance test suite to assert the MPIC invariant across every DCV entry point and every validation method permitted by the CP/CPS, including explicit negative tests for each.	Prevent, Detect	CF #1, CF #3	Test coverage report shows a positive and a negative MPIC test for each DCV entry point and validation method.	2026-11-30	Completed

Action items #4 and #5 are still ongoing and are on track to be completed on or before their due date.

SSL.com continues to monitor this bug for any comments or questions. We ask that the next update field be set to 2026-11-15.