1959867 - Google Trust Services: Inconsistent MPCAA secondary perspective logging

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Google Trust Services]

Preliminary Incident Report

Summary

• Incident description:

Google Trust Services (GTS) is investigating an issue related to logging completeness for a subset (~1.5%) of non-primary perspective MPCAA check results. We have stopped issuance temporarily to speed deployment of our initial mitigation. A full report will be posted within 2 weeks.

• Relevant policies:

TLS BRs section 3.2.2.9 - "Effective March 15, 2025, the CA MUST implement Multi‐Perspective Issuance Corroboration using at least two (2) remote Network Perspectives. The CA MAY proceed with certificate issuance if the number of remote Network Perspectives that do not corroborate the determinations made by the Primary Network Perspective (“non‐corroborations”) is greater than allowed in the Quorum Requirements table."

• Source of incident disclosure:

GTS self-discovered the incident via its automated audit software following a validation service update.

Comment 1

[Google Trust Services]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A004159

• Incident description: Some remote perspective CAA validation results were not completed. ~1.03% of remote perspective MPCAA check results went into an error state that resulted in them not completing and not being logged. MPCAA checks were in a dark launch mode, which meant issuance proceeded despite not all perspectives completing the CAA checks. Failure to complete the checks was due to 2 underlying causes:

1. Some requesting clients had source IP addresses from the non-routable address space (IPv6 fd00:/08), which was unexpected and didn’t pass a data validation check.
2. The DNS resolution thread did not respect request deadlines, causing all remaining work for the CAA check - including audit logging - to be skipped when the deadline was exceeded.

• Timeline summary:

  • Non-compliance start date: 2025-03-15: 00:00 UTC
  • Non-compliance identified date: 2025-04-10: 18:14 UTC
  • Non-compliance end date: 2025-04-11: 03:45 UTC

• Relevant policies: TLS BRs section 3.2.2.9, specifically the recently enforced requirements - "Effective March 15, 2025, the CA MUST implement Multi‐Perspective Issuance Corroboration using at least two (2) remote Network Perspectives. The CA MAY proceed with certificate issuance if the number of remote Network Perspectives that do not corroborate the determinations made by the Primary Network Perspective ("non‐corroborations") is greater than allowed in the Quorum Requirements table."

• Source of incident disclosure: Self-reported. Discovered by GTS as part of self-audit checks.

Impact

• Total number of certificates: 528,586 certificates lacked the desired number of remote perspective details for CAA checks.
• Total number of "remaining valid" certificates: ~160M
• Affected certificate types: This incident affected some DV certificates (OID 2.23.140.1.2.1).
• Incident heuristic: A subset of DV certificates requested by some clients originating in Google Cloud IP address space used non-routable IPs from the fd00::/8 private IPv6 subnet as their source IP. The requesting clients were a mix of Google customers and Google services. Additionally, a subset of DV certificates for which DNS resolution timed out while performing MPCAA checks were also affected. Neither of these properties manifests in a certificate, so this heuristic cannot be used by a third party to assemble the full corpus of affected certificates.
• Was issuance stopped in response to this incident, and why or why not?: Yes. Issuance was stopped for specific stacks during the interval from 2025-04-10 23:31 UTC to 2025-04-11 04:42 UTC to accommodate landing changes faster. Halting also helped provide a clean cutoff to invalidate active authorizations created before issuance was halted to minimize odd states after issuance resumed.
• Analysis: N/A - no revocation was needed.
• Additional considerations: We hope this incident report is useful to other CAs as we hit a number of unexpected edge cases and what was assumed would be a quick and simple fix ended up being far more involved.

Timeline

All times are UTC.

2022-02-02: 12:29 - MPDV is enabled for the first time in GTS's production environment.

2024-12-09 (week of) - A normal progressive rollout dark launched MPCAA for the first time in GTS's production environment. CAA checks from multiple perspectives are attempted for all perspectives, but issuance can proceed in case of failure.

2025-02-14: 18:04 - GTS filed bug #1948368 covering handling of MPIC perspectives by our self-audit tool.

2025-03-15: 00:00 - The issue documented in this report began.

2025-04-03: 20:13 - As part of ongoing improvements, additional auditor checks were submitted.

2025-04-10: 00:34 - The first audit tool report flagging the issue completed and a bug was automatically filed.

2024-04-10: 17:51 - Review and testing to confirm the correctness of the audit tool report was completed and a discussion with the GTS Policy Authority was started about next steps.

2025-04-10: 20:52 - An incident was declared by the Policy Authority.

2025-04-10: 23:31 - Issuance was halted to deploy a fix quickly.

2025-04-11: 02:33 - The preliminary incident report for this issue was filed in Bugzilla.

2025-04-11: 03:45 - The compliance incident ended. (A fix that strictly enforced MPCAA quorum was in place)

2025-04-11: 04:42 - Issuance was resumed.

Related Incidents

Bug	Date	Description
bug 1948368	2025-02-14	This bug covered an implementation error in the GTS self-audit tool related to ensuring non-primary perspective MPCAA records were logged. The new bug covers the discovery and remediation of a logging error within the GTS multi-perspective validation service.

Root Cause Analysis

Contributing Factor 1: Issuance flow sometimes continued even when not all remote perspective CAA checks had completed.

• Description: For all certificate issuances, GTS attempted to perform a multi-perspective CAA check. However, in the case that this check did not complete, issuance could still proceed once the primary perspective found that issuance was permitted as allowed in the March 15th MPCAA rules. This was done to fine tune our MPCAA parameters before full enforcement is required in September, 2025.
• Timeline: 2025-03-15: 00:00 UTC to 2025-04-11: 03:45 UTC
• Detection: Our automated auditing pipeline detected the missing CAA check logs from remote perspectives for a small subset of certificates after new rules were added to the audit tool.
• Interaction with other factors: N/A
• Root Cause Analysis methodology used: 5 whys + fishbone / Ishikawa

Contributing Factor 1.1: DNS timeouts were not handled correctly

• Description: The DNS resolution thread for MPCAA checks did not respect request deadlines, causing all remaining work for the CAA check - including audit logging - to be skipped when the deadline was exceeded.
• Timeline: 2025-03-15: 00:00 UTC to 2025-04-10: 23:31 UTC
• Detection: N/A
• Interaction with other factors: The combination of this and CF #1 that complicated debugging.
• Root Cause Analysis methodology used: 5 whys + fishbone / Ishikawa

Contributing Factor 1.2: Improper handling of unexpected source IP addresses

• Description: We made an assumption in data validation logic, that source IP addresses would always be from routable address space. Google Cloud networking presents some source traffic as coming from the non-routable fd00::/8 space even if the sending service also has a public address. This is possible both for Google run services and customer services running on Google Cloud. While we failed validation for these addresses, it was a silent and unlogged failure as most of our data validation checks are for classes of issues that are typically resolved by a retry. In this case, the issue was persistent and retries were not going to help, so it should have been a fatal error that was propagated. After issuance resumed, while debugging users who were still down, we determined this was also the cause for some of the initial ~1.03% of secondary perspective records that were not recorded and prompted the incident. Some ACME clients that encountered the data validation error failed in ways that made it appear as though the problem was client side, not a server issue. This was especially true for Golang based clients where operations run in a Golang context with timeouts. These failed such that it appeared as though they had timed out, not that they had encountered an error.
• Timeline: 2024-10-16 16:06 (validation code for the source IP added) - 2025-04-11 04:44 (validation code for the source IP adjusted)
• Detection: The initial detection was anomalous results in the internal audit tool report for 2025-04-09, indicating missing entries for some remote perspectives. Connecting that report to this factor required a lot of triangulation and a number of hours.
• Interaction with other factors: This was the root cause of the initial problem. It also caused further challenges after issuance was restored because the sub-set of traffic triggering this problem was concentrated across a small number of services and they were totally down until the root cause was identified and the fix pushed out.
• Root Cause Analysis methodology used: 5 whys + fishbone / Ishikawa

Contributing Factor 2: Insufficient review of dark launch validation results / incorrect logging levels

• Description: When a remote perspective failed to complete due to a data validation error, it was not raised as a fatal error. The issue flew under the radar in general logs until the targeted review for this issue brought the details to light. It is hard to get logging levels perfect, if you elevate the log levels too much, important signals may get lost amongst a high volume or less important details. Conversely if a log level is too low, the issue may remain hidden unless looked for.
• Timeline: 2025-03-15: 00:00 UTC to 2025-04-10: 23:31 UTC
• Detection: N/A
• Interaction with other factors: If dark launch logs had been reviewed in greater detail the error would have been more prominent leading to resolution being faster and easier.
• Root Cause Analysis methodology used: 5 whys + fishbone / Ishikawa

Lessons Learned

• What went well:

• The issue was self-detected.
• After resuming issuance, >99% of requests recovered quickly and without issues.
• MPCAA checks are now enforced by GTS. Quorum is required to issue and we no longer make use of the 'MAY' clause allowing issuance if remote perspectives do not meet the quorum requirements table values.

• What didn’t go well:

• Debugging to establish the root cause took longer than expected.
• We discovered some odd ACME client behavior after implementing our fix that adversely affected a small number of users.

• Where we got lucky:

• The presence of the 'MAY' clause in the 2025-03-15 MPCAA rules prevented the need to conduct mass revocations. (The CA MAY proceed with certificate issuance if the number of remote Network Perspectives that do not corroborate the determinations made by the Primary Network Perspective (“non‐corroborations”) is greater than allowed in the Quorum Requirements table). We seriously considered revoking all affected certificates as we do not want to rely on MAY statements, but in this case we could not justify the impact to clients given the presence of the MAY. MPIC is being phased in via multiple steps to reduce risks to CAs and clients. This was a large learning experience for GTS and we're grateful that MPIC is being deployed in progressive steps to make it easier to focus on specific edge cases and conduct iterative performance tuning.

• Additional:

• N/A

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Ensure issuance cannot proceed with missing MPIC remote perspective data	Prevent	CF #1	Test coverage confirming it is not possible to proceed without all of the required data.	2025-04-11	Complete
Handle DNS timeouts correctly	Prevent	CF #1.1	AI #1 mitigated this issue	2025-04-14	Complete
Adjust IP source validation	Prevent	CF #1.2	Previously failing clients are able to obtain certificates again.	2025-04-11	Complete
Report logging rates by level in dashboards to improve discoverability of changed rates	Prevent	CF #2	The new metrics are present in production dashboarding	2025-05-16	Ongoing

Appendix

N/A

We kindly request that the nextUpdate field be set to 2025-05-16.

Comment 2

[Google Trust Services]

Report Closure Summary

• Incident description: Some remote perspective CAA validation results were not completed. A fraction of remote perspective MPCAA check results went into an error state that resulted in them not completing and not being logged.

• Incident Root Cause(s):

1. Issuance flow sometimes continued even when not all remote perspective CAA checks had completed. (DNS timeouts and unexpected source IPs could both cause problems).
2. Insufficient review of dark launch validation results / incorrect logging levels

• Remediation description: We halted issuance and fixed the initial issue as soon as we confirmed the problem. Test coverage was added to ensure the previous behavior could not recur. Dashboards were updated to have finer grained metrics and highlight changes from recent levels.

• Commitment summary: GTS no longer relies on the 'MAY' behavior allowable until September 15, 2025 when performing MPCAA checks. The proposed action items, which have all been completed, cover all other commitments.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Ensure issuance cannot proceed with missing MPIC remote perspective data	Prevent	CF #1	Test coverage confirming it is not possible to proceed without all of the required data.	2025-04-11	Complete
Handle DNS timeouts correctly	Prevent	CF #1.1	AI #1 mitigated this issue	2025-04-14	Complete
Adjust IP source validation	Prevent	CF #1.2	Previously failing clients are able to obtain certificates again.	2025-04-11	Complete
Report logging rates by level in dashboards to improve discoverability of changed rates	Prevent	CF #2	The new metrics are present in production dashboarding	2025-05-16	Complete

All Action Items disclosed in this report have been completed as described, and we request its closure.

N.B. In the Incident Report filed on 2025-04-24 the 'Related Incidents' section referenced the correct related incident, but it was incorrectly summarized as affecting MPCAA, whereas it was MPIC related. We apologize for this error. It does not meaningfully impact anything from this incident, but we wish to flag the error.

Comment 3

[Antonis]

Dear Google Trust Services,

Thank you for the detailed report! I have a couple of questions to better understand this issue, that I think probably aren't clear in current MPIC requirements, and your use case can help us understand this and maybe improve this policy.

First of all, you completed the "Adjust IP source validation". What does this mean in practice? Did you just remove this check or made it more relaxed? Is issuance now allowed for or when requested by private IP space?

Then, I have some questions about your networking. Currently you say that Google's internal networks, as well as Google Cloud customers' internal networks, are merged. One can use private IP space to reach from one domain to the other. How does this work in practice? Have you done something special because of this to satisfy the NSRs? What is the ownership / scope of Google, of GTS, and of GCP in this single unified network?

With the current NSRs, there's usually a specific point where the CA has full control and authority, and another point where its upstream provider has full control and authority, and these two connect e.g. with a physical cable. For example, a CA edge router or firewall connects to an ISP router. Does this exist here as well?

How is this internal networking functioning in practice? If I rent a GCP Virtual Machine with a public IP address, and I talk to your ACME endpoint, will GCP (or Google? or GTS?) "hijack" (not meant in the bad sense, maybe call it split horizon) DNS and redirect me to GTS' private IPs? Will this be done by routing / NAT? More crucially: when you validate a domain with a 3.2.2.4 section, such as HTTP-01, will this "someone" also hijack DNS or IP traffic and redirect it to fd00::/8 instead of e.g. 192.0.2.1 or 2001:db8::1?

From what I understand from your report, the "primary" perspective (the RA?) will be redirected, so for example.org with DNS records AAAA 2001:db8::1 (a public IP), your CA will (and has been since at least 2022, possibly more) talk to another, different IP address (e.g. fd00::5). The MPIC "secondary" perspectives may use the private IPs (if this perspective also lives within this internal network), or the public IPs (if this perspective lives somewhere else). Is this a factor of why the percentage is only ~1.5%?

If I am talking to GTS, e.g. its ACME endpoints using HTTPS, or OCSP / CRLs, is this terminated by Google, and then forwarded to GTS' private IPs? Or are the public IPs configured directly on a server's network interfaces within GTS' control? In other words, does Google (or GCP?) apply Destination NAT / Port Forwarding, do they terminate TCP, do they terminate TLS, do they terminate HTTP?

Apologies for the many technical questions, I think it's important to understand the path of incoming and outgoing traffic and its various types (domain validation, OCSP, DNS, ACME). With the Cloud, networking lines have been blurred, and perhaps MPIC as it's standardized today, may not address every scenario as intended. The reason I ask you is because if another CA used GCP (or AWS? Azure?) for either MPIC or RA or similar components, they could have been using these private IPs too without ever being able to know this is happening, when a customer in that Cloud requested a certificate.

Again, thank you very much, and I look forward to your response!

Comment 4

[Google Trust Services]

Thank you for the questions, Antonis. Before we address your questions below, we think it’s important to clarify relevant facts.

This incident occurred because GTS did not complete remote perspective MPCAA checks for all issuances. The reasons this happened are twofold:

The DNS lookups for some remote perspective MPIC checks timed out.
ACME requests from a private IP address space to the service failed data validation before even starting the remote perspective MPCAA process.

Both of these are software bugs, not compliance violations. The issue is how our systems responded to these bugs; failing open instead of failing closed. GTS’s ACME endpoint can receive requests from the Google Cloud network, but GTS’s MPIC and DCV systems are not implemented on the Google Cloud network.

Other CAs are able to use Google Cloud or any other Cloud provider to implement MPIC because BR Section 3.2.2.9 states that computing systems performing MPIC are considered outside of the CA’s audit scope with a few explicit exceptions.

With this in mind, responses to the specific questions are inline.

(In reply to Antonis from comment #3)

Dear Google Trust Services,

Thank you for the detailed report! I have a couple of questions to better understand this issue, that I think probably aren't clear in current MPIC requirements, and your use case can help us understand this and maybe improve this policy.

First of all, you completed the "Adjust IP source validation". What does this mean in practice? Did you just remove this check or made it more relaxed? Is issuance now allowed for or when requested by private IP space?

We added logic to handle non-routable source addresses in the system performing remote perspective MPCAA checks in a manner that fails closed. Issuance has never been allowed for certificates whose subject name or SANs include IPs in the private IP space.

CAA checks from the primary perspective did not experience DNS timeouts or failures due to unexpected source IPs and completed correctly.

Then, I have some questions about your networking. Currently you say that Google's internal networks, as well as Google Cloud customers' internal networks, are merged. One can use private IP space to reach from one domain to the other. How does this work in practice? Have you done something special because of this to satisfy the NSRs? What is the ownership / scope of Google, of GTS, and of GCP in this single unified network?

Google’s internal networks and Google Cloud customers’ internal networks are not merged.

In this case, GCP projects using the Google Cloud product VPC Service Controls are able to access our public ACME endpoint at https://dv.acme-v02.api.pki.goog/directory, like any other requester.

With the current NSRs, there's usually a specific point where the CA has full control and authority, and another point where its upstream provider has full control and authority, and these two connect e.g. with a physical cable. For example, a CA edge router or firewall connects to an ISP router. Does this exist here as well?

Yes, this exists in GTS’s CA Infrastructure.

How is this internal networking functioning in practice? If I rent a GCP Virtual Machine with a public IP address, and I talk to your ACME endpoint, will GCP (or Google? or GTS?) "hijack" (not meant in the bad sense, maybe call it split horizon) DNS and redirect me to GTS' private IPs? Will this be done by routing / NAT? More crucially: when you validate a domain with a 3.2.2.4 section, such as HTTP-01, will this "someone" also hijack DNS or IP traffic and redirect it to fd00::/8 instead of e.g. 192.0.2.1 or 2001:db8::1?

GTS is not familiar enough with the details of how Google Cloud Products direct traffic to its ACME endpoint to comment on this. This is outside of the CA boundary and comparable to optimizations a customer's CDN or managed network provider may make.

The private IPs referred to in this incident are IPs controlled by customers and not GTS. GTS received requests from these customer IPs on its ACME endpoint.

No DNS or IP traffic is 'redirected' when GTS performs domain validation.

From what I understand from your report, the "primary" perspective (the RA?) will be redirected, so for example.org with DNS records AAAA 2001:db8::1 (a public IP), your CA will (and has been since at least 2022, possibly more) talk to another, different IP address (e.g. fd00::5). The MPIC "secondary" perspectives may use the private IPs (if this perspective also lives within this internal network), or the public IPs (if this perspective lives somewhere else). Is this a factor of why the percentage is only ~1.5%?

The preliminary report was based on a heuristic and put the affected certificate population at ~1.5%, but the precise number was 1.03%, which was corrected in the full report.

The percentage of 1.03% was the combined impact of DNS timeouts and non-routable source addresses. Within that 1.03%, 1.01% of the errors were due to DNS timeouts and 0.02% were because the requests originated from a private IP space.

If I am talking to GTS, e.g. its ACME endpoints using HTTPS, or OCSP / CRLs, is this terminated by Google, and then forwarded to GTS' private IPs? Or are the public IPs configured directly on a server's network interfaces within GTS' control? In other words, does Google (or GCP?) apply Destination NAT / Port Forwarding, do they terminate TCP, do they terminate TLS, do they terminate HTTP?

The private IPs mentioned in this incident report refer to private IPs belonging to GTS customers and not GTS.

Apologies for the many technical questions, I think it's important to understand the path of incoming and outgoing traffic and its various types (domain validation, OCSP, DNS, ACME). With the Cloud, networking lines have been blurred, and perhaps MPIC as it's standardized today, may not address every scenario as intended. The reason I ask you is because if another CA used GCP (or AWS? Azure?) for either MPIC or RA or similar components, they could have been using these private IPs too without ever being able to know this is happening, when a customer in that Cloud requested a certificate.

GTS’s MPIC system is not implemented using GCP. The private IPs referred to in this incident belong to GTS customers, not GTS. If another CA used GCP for MPIC or RA or similar components, they could not have been inadvertently using these private IPs. We do not see specific new challenges here, nor do we see a need for MPIC requirement changes. If there is a community learning based on this question, it is probably for each CA to think about how Postel's Law ("be conservative in what you send, be liberal in what you accept") applies to their setup.

Again, thank you very much, and I look forward to your response!

We believe this addresses all the open questions. We will continue to monitor this bug.

Comment 5

[amir]

Thank you for your detailed answer!

If I am talking to GTS, e.g. its ACME endpoints using HTTPS, or OCSP / CRLs, is this terminated by Google, and then forwarded to GTS' private IPs? Or are the public IPs configured directly on a server's network interfaces within GTS' control? In other words, does Google (or GCP?) apply Destination NAT / Port Forwarding, do they terminate TCP, do they terminate TLS, do they terminate HTTP?

The private IPs mentioned in this incident report refer to private IPs belonging to GTS customers and not GTS.

I don't think this sufficiently answers the question posed. For what it's worth, I don't think https being enabled or not is a major problem for ACME CA servers as the protocol has other built-in protections. Despite that, I think part of the question that was left unanswered can be summarized "Which entity terminates the TLS connection for the Google Trust Services ACME endpoint? Is it Google Trust Services (the CA), or Google (not the CA), or some other entity?"

More broadly, are there any entities outside of Google Trust Services that have access to the private key for the https endpoint of the CA services?

Comment 6

[JR Moir]

Thank you GTS for a detailed report.

Can you confirm you are now enforcing MPIC in the way required after Sept 15 by all other CAS?

Comment 7

[Google Trust Services]

Thanks for the comment, Amir. Response below.

(In reply to amir from comment #5)

Thank you for your detailed answer!

If I am talking to GTS, e.g. its ACME endpoints using HTTPS, or OCSP / CRLs, is this terminated by Google, and then forwarded to GTS' private IPs? Or are the public IPs configured directly on a server's network interfaces within GTS' control? In other words, does Google (or GCP?) apply Destination NAT / Port Forwarding, do they terminate TCP, do they terminate TLS, do they terminate HTTP?

The private IPs mentioned in this incident report refer to private IPs belonging to GTS customers and not GTS.

I don't think this sufficiently answers the question posed. For what it's worth, I don't think https being enabled or not is a major problem for ACME CA servers as the protocol has other built-in protections. Despite that, I think part of the question that was left unanswered can be summarized "Which entity terminates the TLS connection for the Google Trust Services ACME endpoint? Is it Google Trust Services (the CA), or Google (not the CA), or some other entity?"

Google Trust Services leverages Google infrastructure to terminate TLS using an automated process.

More broadly, are there any entities outside of Google Trust Services that have access to the private key for the https endpoint of the CA services?

The private key of the HTTPS endpoints only exists in a hardened process. No person, regardless of whether they are affiliated with GTS, Google, or any other entity, can access the private key.

GTS would like to emphasize that this compliance issue was caused by failures of remote perspective MPCAA completion. We're happy to answer community questions unrelated to this incident in another forum.

Comment 8

[Google Trust Services]

(In reply to JR Moir from comment #6)

Thank you GTS for a detailed report.

Can you confirm you are now enforcing MPIC in the way required after Sept 15 by all other CAS?

MPIC encompasses MPCAA and MPDV. GTS is enforcing MPCAA for all requests as required by the September 15, 2025 effective date.

GTS has enabled MPDV on our public endpoint since 2022. We have always used 5 remote perspectives, greater than the 2 remote perspectives required on September 15, 2025. To date, we have allowed 2 non-corroborating perspectives - this is acceptable currently - but effective Sep 15, 2025 only 1 non-corroboration is allowed. GTS does not foresee any issues meeting future MPDV requirements by September 15, 2025.

We believe this addresses all the open questions. We will continue to monitor this bug.

Comment 9

[Google Trust Services]

GTS is continuing to monitor this bug.

Comment 10

[Google Trust Services]

All questions have been answered and there have not been any additional follow-up questions. GTS requests closure of this bug. Thank you.

Comment 11

[incident-reporting]

A report summary was filed on May 15, 2025.

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-06-10.

Comment 12

[Google Trust Services]

GTS is continuing to monitor this bug, pending closure.