1948368 - Google Trust Services: Self-audit tooling MPIC perspective verification inconsistency

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Google Trust Services]

Google Trust Services is investigating a deviation in the behavior of our self-audit tooling used to satisfy section 8.7 of the BRs. The deviation is related to how it verified MPIC perspective details following certificate issuance. At this time we do not believe there was any mis-issuance or failure to perform MPIC in the manner indicated by our CPS. The problem appears to be isolated to a failure by the self-audit tooling to always check the primary perspective audit entry. In some cases, the tool could have checked the corroborating perspective audit entry instead. We have fixed the incorrect code and are working to verify the correctness of all records since MPIC was enabled. A full incident report will be provided by 2025-02-25.

Comment 1

[Google Trust Services]

Incident Report

Summary

Google Trust Services maintains a custom self-auditing tool used to satisfy section 8.7 of the BRs. The self-audit tool reconciles multiple sets of issuance records and performs verifications to confirm adherence to the applicable BRs and our CP / CPS.

A bug in the tool resulted in it using the first successful validation record found and, regardless of whether it was the primary perspective or a corroborating one, accepted it as the authoritative perspective for auditing purposes. The issuance process, separate from the self-auditing tool, has always required a successful primary perspective to validate, but a corroborating perspective was incorrectly accepted by the self-auditing tool as the primary perspective in some cases.

The bug was detected while reviewing the code in preparation for MPIC requirements, which become effective on March 15, 2025. A patch was submitted ~2 hours after detection. GTS ran a historical analysis using the patched self-audit tool of the logs from the past two quarters to confirm there was no improper validation or mis-issuance related to this bug. GTS also ran historical checks against records for the last two years to verify the validations were done correctly. Neither run reported any issues.

Impact

There was no impact to issued certificates. No revocations were necessary.

Timeline

All times are UTC.

2018-08-27:
18:40 Domain validation verification logic added to the self-auditing tool.

2020-05-18:
17:04 CL updating the audit logs to allow multiple perspectives is submitted.

2022-02-02:
12:29 MPDV, now known as MPIC, is enabled for the first time in production, effectively introducing the bug in the self-auditing tool

2025-02-12:
18:30 An engineer working on adding support for new MPIC requirements to the self-auditing tool notices the bug and kicks off the incident management process.
18:30 Formal investigation and analysis of historical certificate data begins.
20:19 A change is submitted which fixes the bug in the self-audit tool code and adds tests exercising the issue.
21:58 The GTS Policy Authority officially declares this a public incident.

2025-02-14:
19:04 GTS publishes a preliminary report to Bugzilla.

2025-02-19:
00:22 GTS finishes rerun using the patched self-audit tool on the past 60 days of active certificates and finds no violations.
23:00 GTS finishes rerun using custom validation on the past 2 years of certificates and finds no violations.

Root Cause Analysis

Background

For each issuance, the self-auditing tool collects audit log entries for all validations associated with that issuance. It then iterates through the entries, validates them, and adds each correctly validated domain to a set. Finally, it verifies that this set of domains matches the set of domains in the issued certificate.

Before MPIC was implemented, there was one audit log entry generated per validated domain. The above logic correctly validated the domain validation procedure.

After MPIC was implemented, each perspective generated an audit log entry. There were 6 log entries generated per validated domain. The original logic was not updated. When iterating through the entries, it added a domain to the validated set if any one of the 6 entries correctly validated it without limiting it to the primary perspective.

Self auditing tool not updated when audit logs were updated

The key oversight that caused this bug was that GTS updated the format of the audit logs, but didn’t simultaneously update the self-auditing tool which used the audit logs as input. GTS was an early adopter of MPIC (named MPDV at the time) as an experiment. It took 3+ years to operationalize our implementation of MPIC, which took place some time before relevant requirements were adopted into the Baseline Requirements. Given the experimentational nature of MPIC at the time, the need for corresponding changes to the self-audit tool were not recognized. Going forward, GTS is adding an automated submission check when log format changes are made to ensure the self-audit tool is also updated when necessary.

Lessons Learned

What went well

• GTS was able to quickly patch and deploy a change upon detection.

• Defense in depth protected issuance.

• As part of implementing new MPIC requirements, GTS detected the bug.

What didn't go well

• It took longer to run the longer term data analysis than expected due to growth in the audited dataset.

Where we got lucky

• The issue was self detected and limited to a deviation between our detection system and actual behavior, which followed the requirements.

Action Items

Action Item	Kind	Due Date
Fix the bug in the self-auditing tool that resulted in this incident	Prevent	2025-02-14 (Complete)
Document how GTS made the self-audit tool’s run time faster for future events that require data analysis	Mitigate	2025-02-17 (Complete)
Verify past updates to the audit log format from the past 2 years do not require corresponding updates to the self-audit tool	Detect	2025-03-21
Add an automated submission check for audit log format changes to ensure the self-audit tool is also updated when necessary	Mitigate	2025-03-21

Appendix

Details of affected certificates

N/A

Comment 2

[Google Trust Services]

GTS has completed the remaining two Action Items: “Add(ing) an automated submission check for audit log format changes to ensure the self-audit tool is also updated when necessary” and “Verify(ing) past updates to the audit log format from the past 2 years do not require corresponding updates to the self-audit tool”.

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Fix the bug in the self-auditing tool that resulted in this incident	Prevent	Self auditing tool not updated when audit logs were updated	GTS added test cases that fail when the bug is present	2025-02-14	Complete
Document how GTS made the self-auditing tool’s run time faster for future events that require data analysis	Mitigate	It took longer to run the longer term data analysis than expected due to growth in the audited dataset	GTS will conduct analysis speed tests as part of each annual WebTrust audit	2025-02-17	Complete
Verify past updates to the audit log format from the past 2 years do not require corresponding updates to the self-audit tool	Detect	Self auditing tool not updated when audit logs were updated	A comprehensive review of all changes from the past 2 years was conducted and recorded that there were no additional issues	2025-03-21	Complete
Add an automated submission check for audit log format changes to ensure the self-audit tool is also updated when necessary	Prevent	Self auditing tool not updated when audit logs were updated	Submission checks will ensure that changes to the audit log format are also updated in the self-audit tool and record an attestation for each code change submission	2025-03-21	Complete

GTS is providing the updated list of Action Items above using the new table format from the CCADB Incident Reporting Guidelines that went into effect 2025-03-01, so the new columns have been added since our previous comment.

GTS will continue to monitor this bug for comments or questions and will send a closure report on Friday, March 7th, 2025 if none are received.

Comment 3

[Google Trust Services]

Report Closure Summary

• Incident description: A bug in GTS’s self-auditing tool resulted in the incorrect verification of the MPIC perspectives used in domain control validation. The bug occurred due to using the first successful validation record found, regardless of whether it was the primary perspective or a corroborating one, accepting it as the authoritative perspective for auditing purposes.
• Incident Root Cause(s): The key oversight that caused this bug was that GTS updated the format of the audit logs but didn’t simultaneously update the self-auditing tool which used the audit logs as input.
• Remediation description: GTS has fixed the bug and taken steps to ensure there are checks in place to reduce the risk of this type of bug occurring in the future. GTS also made improvements to its data gathering tools to improve the speed of analysis for future incident responses.
• Commitment summary: GTS continues to identify opportunities to reduce the likelihood of missing updates between dependent components.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 4

[Ben Wilson]

I will review this bug on Wed. 12-Mar-2025 and intend to close it at that time, unless there are any comments or discussions that need to be addressed.

Comment 5

[Google Trust Services]

Google Trust Services continues to monitor this bug for comments or questions. If there are none, we kindly request its closure.