1876593 - Google Trust Services: Failure to properly validate IP address

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Google Trust Services]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

Steps to reproduce:

Google Trust Services is investigating an IP validation issue that occurred when validating a single IP address. All affected certificates have been revoked.

We will post a full incident report by Tuesday of next week.

Comment 1

[Google Trust Services]

Attachment: revoked_certs.txt

Comment 2

[Google Trust Services]

Attachment: all_affected_certs.txt

Comment 3

[Google Trust Services]

Incident Report

Summary

Background

Google Trust Services’ (GTS) earliest iteration, Google Internet Authority (GIA) relied upon EJBCA and a series of scripts to issue organization validation (OV) certificates for “Alphabet-owned” domains and IPs. Validations were stored in flat files as early as 2011. The project was run by a few security engineers (aka certificate authority engineers or CAEs) and technical program managers. As GTS expanded, the need for automation was apparent and a different team of software engineers developed an ACME-based service to offload the work from the CA Engineering team. Since the original validations were manual and OV-based, many clients lacked support to dynamically solve domain validation (DV) challenges. As a path forward, the ACME-based infrastructure was allowed to support the existing validation flat files with additional ACL checks via internally defined challenges. This decision made it easy to onboard the existing clients onto “ACME”, but it didn’t relieve the CA Engineering team completely as they still managed the validations on the backend. The ACME service launched in 2018.

The ACME infrastructure was eventually extended to support all of the standard DV challenges defined by the IETF for both domains and IP addresses. The service grew from a few thousand certificates with a one-year validity period to millions of 90 day certificates. By the end of 2021, the old OV validation flat files were simplified to be DV validation flat files, but the team was unsuccessful in migrating all of the old clients over to ACME DV challenges. There were a number of hurdles including:

• Many of Google’s sites have availability SLOs of five nines where quick global changes for solving ACME challenges and pushing new certificates to infrastructure are not directly compatible. Special ACME client logic is necessary.

• Existing clients relied heavily on parent domain authorization applying to many subdomains and ACME support for these authorizations was in flux. There is now RFC 9444 [1], but complexity increases regardless for both client and server.

• Devices and services were reliant on publicly trusted certificates where private PKI may be a better fit if redesigning the system. For example, corporate printers. These services have required special ACME integrations or resources to move.

• Clients became accustomed to the legacy methods. From the client's perspective, issuance worked and there were few incentives for each individual team to spend resources to further migrate.

The stored validations are separated into two files, one for domains and another for IP addresses. While the validation file with stored domains received tooling and investment to automate as much as possible given the frequency with which it was touched; the IP file is rarely modified and did not get the same treatment. Only one IP has been added to the file in the past year and that single addition is the subject of this incident.

IP Address Validation Flat File

GTS utilizes BR 3.2.2.5.3 [2], “Reverse Address Lookup”, to validate the stored IP address validations for Alphabet-owned IPs which have not transitioned to the RFC 8738 ACME IP address challenges [3]. The procedure itself requires multiple steps. CAEs validate the IP addresses through a series of manual commands. BR 3.2.2.5.3 validates using dig with Google DNS to resolve the IP’s associated PTR record, this does not harness the protections in place for the ACME DVS service as described in https://bugzilla.mozilla.org/show_bug.cgi?id=1873739#c10. The resultant domain is checked via BR 3.2.2.4.7 [4] using an in-house binary that automates the validation process and ultimately proposes changes to the domain validation flat file. If the process succeeds, the engineer writes the output of both processes in a log. GTS acknowledges there are many areas of improvement in this routine, but we do believe it meets the minimum requirements set by the Baseline Requirements and our CP/CPS.

After completion of the IP validation routine, the validations are stored in a multi-party authorization (MPA) protected flat file. Each validation contains the identifier, the date when the validation was performed, the BR method employed, and a link to the associated evidence.

All stored validations are targeted to be renewed every 3-6 months. Teams owning the identifiers that do not pass re-validation are contacted in an attempt to resolve issues. If they are unable to fix the problem, the identifiers are removed.

The scale of the procedure pales in comparison to GTS’ other issuance. There are less than 50 IP addresses in the validation flat file. Certificates including identifiers validated using this method represent ~0.0015% of unexpired GTS certificates.

GTS attempts to audit all ACME issuance and validations daily for possible violations. For example, each issuance log is matched to appropriate validation, authorization, CAA, and request logs. In the case of the internal challenges though, it treats the audit log details about the flat file as ground truth.

The validation flat files do not have automated auditing to verify that each validation entry has associated evidence.

Incident

In March 2023, the IP address of 64.154.127.30 was proposed by a non-CAE as an addition to the IP validation flat file and was approved by a CAE without the submission of validation evidence, violating BR 5.4 [5]. The link referenced in the validation record included the addition of the IP address, but no validation evidence was added.

Although command history indicates that the CAE executed the validation procedure, the results were not formally captured and GTS cannot confirm the correct action was taken.

At the end of June, the periodic validation procedure identified that 64.154.127.30 was failing validation. The IP was removed and the associated contact was notified. They did not need the service yet so the records were not fixed. Two additional certificates were issued between detection of the issue and the removal of the IP from the flat file. GTS did not consider this a problem as it was believed to be a temporary misconfiguration on a pre-production service. The original authorization was still believed to be valid.

In December, the same team sent a proposal to re-add 64.154.127.30 with the original validation date specified in March, and the proposal was again accepted by the CAE. Technically, this adheres to the requirements as it was reusing the prior validation, but it is atypical to how GTS generally handles Alphabet-owned IP validations. There were no additional logged conversations for this reinsertion.

In January 2024, a second CAE was working to automate the IP address validation procedure and discovered that the link to the evidence for the IP address 64.154.127.30 did not contain the necessary information for auditing. The validation record was removed from the list again and given that we could not rely upon the validation, all 12 associated certificates were revoked within 24 hours following BR 4.9.1.1 [6].

The reverse lookup of the offending IP address at the time of discovery contained:
30.127.154.64.in-addr.arpa. 3600 IN PTR GOOGLE-INC.bar1.KansasCity1.Level3.net.

Level3 controls the reverse zone file for that address space. We do not have access to its history to know how the PTR records have been updated over time. The subscriber has fixed their record and can now prove control over the associated IP.

30.127.154.64.in-addr.arpa. 3600 IN PTR 30.127.154.64.in-addr.ssh-fe.goog.

Impact

There were 40 certificates issued utilizing the validation in question for 64.154.127.30 from April 10th 2023 to January 22nd 2024. 12 were active at the time of incident discovery and all were revoked within 24 hours. All certificates were short-lived certificates with lifetimes shorter than 45 days.

Timeline

All times are UTC.

2011-02-06

• 11:38 Domains validation flat file is introduced.

2018-03-19

• 13:19 GTS’ ACME for OV certificates launches.

2018-05-08

• 13:38 IPs validation flat file is introduced.

~2019-09-10

• ACME for DV certificates is launched.

~2020-02-01

• RFC 8738: ACME IP Identifier Validation Extension is published.

~2021-10-29

• OV domains validation flat file is replaced by a DV domains validation flat file.

2023-03-28:

• 13:52 Validation for 64.154.127.30 added to validation flat file with incomplete audit logs.

2023-04-10:

• 02:03 First certificate containing 64.154.127.30 issued https://crt.sh/?id=9110346295.

2023-06-30:

• 18:53 Periodic validation attempt reveals that 64.154.127.30 has invalid PTR record.

2023-07-03

• 02:03 Two additional certificates containing 64.154.127.30 are issued.

2023-07-06:

• 13:35 Validation for 64.154.127.30 removed from validation records.

2023-07-07:

• 04:21 Periodic validation for all remaining IPs with stored validations is completed.

2023-11-16:

• 17:26 Periodic validation for all stored IP validation records is performed.

2023-12-07:

• 18:12 Validation for 64.154.127.30 re-added to validation records with the validation timestamp from the original March submission.

2023-12-11:

• 02:02 First certificate containing 64.154.127.30 issued after the validation is reinserted into the flat file.

2024-01-24:

• 14:30 Review indicates that validation evidence is missing.
• 15:17 Manual validation reveals that the PTR record doesn't point to a validated FQDN.
• 15:20 Validation for 64.154.127.30 removed from validation records.
• 16:48 All but one (210.202.247.61) stored IP authorizations are re-validated successfully. We have a valid authorization record from 2023-11-16 for that one IP and we are working with the subscriber to fix their record.
• 19:31 GTS declares that this is an incident.

2024-01-25:

• 00:51 All active certificates are revoked.
• 10:09 A second review confirms that all active certificates are revoked.

2024-01-26:

• 16:28 MPA configuration for validation flat files is fixed to require 2 CAEs to approve changes.

2024-01-29:

• 08:18 Our internal integration documentation now instructs users to open a ticket instead of proposing the change directly.
• 08:19 A prompt was added which requires CAEs to confirm that the validation procedure was followed prior to approval.
• 08:29 The Alphabet-owned IP address validation procedure now includes callouts reminding the reader to record validation evidence.
• 15:33 All historical additions to the IPs validation flat file are verified to ensure they were properly validated and recorded.

2024-01-30:

• 05:03 GTS validates 64.154.127.30 is controlled by the original subscriber.
• 20:16 All historical periodic reviews of the IP validation flat file are verified to ensure they were properly validated and recorded.

Root Cause Analysis

Weak Technical Controls

The IP validation flat file relied upon CAE training and unattached documentation for its correct handling. There were few technical controls to guarantee the procedure was followed including:

• Lack of automation - It would obviously be best if all clients utilized the implemented ACME service, but barring that, any sort of CAE procedure should involve a total of one step. All intermediate steps should happen automatically and not involve any human judgment.

• No automated audits - The problem could have been detected before misissuance if there was any kind of automated secondary check on the stored validation.

• MPA - MPA was intended to be enforced for the file, but it was not configured correctly; it only required one CAE and any other Alphabet employee. Non-CAEs should not be expected to know the requirements. Two CAEs should have been required for changes to increase the likelihood of a correct result given the file’s importance.

• No checklist confirmation - Awareness would have been raised if the system would have prompted the CAE to confirm they followed all steps of the procedure prior to approving the changes.

Edge Case

Processes and code that are utilized infrequently tend to break. The offending IP address was the only one added to the file in all of 2023. Each branch of logic adds considerable cost to a project and the benefits need to outweigh the costs.

Prioritization

• Perfect is the Enemy of Good - The team has spent resources on attempting to help teams migrate to IETF ACME challenges rather than ensuring the review and safe execution of the current long standing processes. Complete migration to IETF ACME challenges is a much larger project which has spanned several years.

• Perceived Impact - The IP validation flat file is modified so rarely and is used for so few certificates, that its cleanup was not highly prioritized. Validation is one of the primary roles of a CA and any failure is unacceptable.

• Siloed Roles - GTS has historically been composed of two teams: (1) a software engineering team and (2) a team of CA Engineers and technical program managers originally founded during the GIA-era. The two groups worked on disjointed projects for many years, and have not fully utilized each team’s strengths in all cases, especially on historical projects.

Lessons Learned

• There is a high cost to infrequently used features; we must either invest the resources to maintain them properly or drop them from the portfolio.
• GTS needs to better utilize resources across teams.
• All validation processes should be centrally located for conformity and best practices.

What went well

• GTS detected the issue while working on automating the process.
• The affected certificates were quickly revoked.

What didn't go well

• The IP validation procedure was not followed correctly - failure to capture the output.
• The IP address was re-added to the validation flat file with the original validation date, but given the unusual circumstances it should have prompted further investigation.
• The problem went undetected for several months. There was no automated system to detect infractions.

Where we got lucky

• IP address revalidation process helped detect the non-compliance.
• The subscriber was able to prove control of the IP address after revocation.
• The affected certificates were all short-lived. Each was issued with less than 45 days of validity. The current maximum lifetime for the deployed configuration is 91 days.

Action Items

Action Item	Kind	Due Date
Update validation flat files MPA settings to require 2 CAEs	Mitigate	2024-01-26
Ensure no other IP flat file validations are missing audit evidence	Detect	2024-01-29
Require CAEs to confirm they followed the procedure when updating validation flat files	Mitigate	2024-01-29
Update the validation procedures to emphasize that CAEs must record validation evidence	Mitigate	2024-01-29
Ensure all validations use GTS’ domain validation service	Prevent	2024-03-01
Automate the proposal of updates to the IPs validation flat file	Prevent	2024-03-01
Automate the archival of validation evidence	Prevent	2024-03-01
Team review of all historical projects & processes	Detect	2024-03-01
Set permissions of the validation records to only be writable by the role running the automated process	Prevent	2024-03-08
Design & implement continual compliance for validation flat files	Detect	2024-05-01

We believe all identified root causes are addressed through these AIs. There are other AIs that will take a significant amount of time to fully rectify, e.g. completely eliminating the validation flat files. We will be unable to commit to a firm deadline for these at this time.

[1] https://www.rfc-editor.org/rfc/rfc9444.html
[2] https://github.com/cabforum/servercert/blob/main/docs/BR.md#32253-reverse-address-lookup
[3] https://datatracker.ietf.org/doc/html/rfc8738
[4] https://github.com/cabforum/servercert/blob/main/docs/BR.md#32247-dns-change
[5] https://github.com/cabforum/servercert/blob/main/docs/BR.md#54-audit-logging-procedures
[6] https://github.com/cabforum/servercert/blob/main/docs/BR.md#4911-reasons-for-revoking-a-subscriber-certificate

Appendix

Details of affected certificates

Revoked Certificates (12 total)
revoked_certs.txt

sha256=55E421E304B6A11F8FA7665E9160B42D791F9A7EE6B5D97824309650F7944A5D
sha256=B2BA76E9BCCA0046D55477F99B56C2ECDAB04BB998585956ED433AFDCA38D8EB
sha256=F75864B5791A3DA1F4012180A29896E48F95C4A95F10E863B0436806A981180E
sha256=0C0C8A3E5C0D934F52B105EE295D162C1DB1704904A7B2D90D1932E20486A517
sha256=63EE2960FB4D9C66D8AFB0E50BC8B6614ED6AD5F1B8E49253B4925BBA09BC008
sha256=B27C34A2A2B1815C7641045E51517D35CE4F3E00FE93F5ECD3BB57401B8A0DB1
sha256=9D3A7FEEE1BEA32810B67A1C358A3189F9E16586B5CA57D64ECB0020173B4E3F
sha256=EE01336270170233BFA792C8209A8C4053A7F8E74D1E7FDF133DDC29912A1BC2
sha256=9B94EFF6A466D1650808D4F30A9A2B06FE5FF3A6CC036CD0B3AFA28A27B4E727
sha256=D58E055E2578AA5042D8D3153CDF0B1B647F5F1CC7224971EFAB939FCF8A63E9
sha256=F9C8AAFAC8DC24FD2F73C55CE1FAADF6D34B151C3206C07406ABEDED2C6A2AEB
sha256=FE378128120EEF7F14FC54E36CA6645194E7D93AE734DE0031445A538C1ECEE5

Complete list of certificates in scope (40 total)
All_affected_certs.txt

sha256=5CF94B8BE30A50979272CB7FD0ED92CB0A573C56AED859D245BEE8B213416821
sha256=E6AE2275228FC42CE8095DFB880056F4754E6F638DB08D3987B0F06C0B70B88B
sha256=FAB8A51A898281B4268063D2E699C5A1D87D2B3619A251B5AF8C2CFDC9C393B8
sha256=7144BEFBDE77A0B2CBDA0379AB7990DF70BDF5AD4DAB611407A5C6AE41ABBA04
sha256=3A780372E46A6D6248F30CE7EF142B41F61E095F43F5A60290A03AB3B157893C
sha256=5097559C2C62AA3685C90D3608C3E93C5437A8923E318FA4F518279352B39695
sha256=A2DFD6B5142E42898B5B615BDCDE45351FF833FC944FD1EBF31F9B59B0C20028
sha256=88BBE2C6A38DF622932A2E146F3E511530A8387B1582A774D259BA596BB1C1C7
sha256=F46549A3E5D60C5FBF0ABA3D22AF203B8EEB72D9433841882EED03573EA077BC
sha256=11EE15B3BFF7A7D41CA80F775BF7D18E90296AE40BE744ED11F532B94390D46B
sha256=F43FB58658EF75B4933FDDB94DA7365E2117F814D09FC60061735B9EDCEBF9B2
sha256=244ABE8B47629AF3F4F2EE5A9D18712F6DF685FF9F4782C02E6F637E6890CD88
sha256=1BF836CB1EB8EFEDB7E65ADF28AF0483F7493D7C8915479DC70E24135DA7E283
sha256=80E73F834475215E9F912D716F3D97F995609DF02E4041C7B9868AB54F91961B
sha256=5B5D00DE457628D91A46E6AA28D9A3C8FB7D35C012CB32D7BC1DF1FDA741ACC6
sha256=00FE5524D505D2BDCA0DC143F2772DF3EBC2C04339BDDF0A5858B1C9D47F97D8
sha256=AEB02AAF91E2C4894E67C9472D218F1D894F8CC0BFB1E4EA2B297375C979BD9D
sha256=ABA5526D55A56C96EF532A74059FDD5F937AC810F49DC2FFBB905BEDB10D5098
sha256=EACBE819DA0F150C9210CACB615F684939B8B2D7BD51E54D89501FC9EF777699
sha256=F8E72B5C4E0F9C6B550F613DC77FB05E4C5D3C024AE67145C1E7691F6A2FACF6
sha256=7E73494DDD8D120F0060C54520192AB1393AEBF8F46FB533F4883C4F20528062
sha256=4676948AD057B4C5C717F70EC8DA42B3102273AD1D32A7075DCA99A92472D76A
sha256=466FEB78AFB3987C16382C2CBFD432E64ABFC0DD076BCE9B19F76F28639F37BF
sha256=31561596512C74D9F650198D289693E3FB2C166B7D22671DFFF37C66D498CE56
sha256=7064D5AF18FC9E9B25C0D49D589A18C912E2A5936815AC44410CCDD294B59822
sha256=CB294093C7884153AB0C3BC1BA29B67F38AE1575A49894456DC3974975DA4CDA
sha256=A93F8FA3E24A6AC049644DB0C12FB27F217B95A84811F086748C7671223822FE
sha256=1A08A9A37D91DA1B13A724D3CD63BA1164B08FE1E2CF0E691EF594E5856310E9
sha256=55E421E304B6A11F8FA7665E9160B42D791F9A7EE6B5D97824309650F7944A5D
sha256=B2BA76E9BCCA0046D55477F99B56C2ECDAB04BB998585956ED433AFDCA38D8EB
sha256=F75864B5791A3DA1F4012180A29896E48F95C4A95F10E863B0436806A981180E
sha256=0C0C8A3E5C0D934F52B105EE295D162C1DB1704904A7B2D90D1932E20486A517
sha256=63EE2960FB4D9C66D8AFB0E50BC8B6614ED6AD5F1B8E49253B4925BBA09BC008
sha256=B27C34A2A2B1815C7641045E51517D35CE4F3E00FE93F5ECD3BB57401B8A0DB1
sha256=9D3A7FEEE1BEA32810B67A1C358A3189F9E16586B5CA57D64ECB0020173B4E3F
sha256=EE01336270170233BFA792C8209A8C4053A7F8E74D1E7FDF133DDC29912A1BC2
sha256=9B94EFF6A466D1650808D4F30A9A2B06FE5FF3A6CC036CD0B3AFA28A27B4E727
sha256=D58E055E2578AA5042D8D3153CDF0B1B647F5F1CC7224971EFAB939FCF8A63E9
sha256=F9C8AAFAC8DC24FD2F73C55CE1FAADF6D34B151C3206C07406ABEDED2C6A2AEB
sha256=FE378128120EEF7F14FC54E36CA6645194E7D93AE734DE0031445A538C1ECEE5

Comment 4

[Google Trust Services]

GTS is monitoring this bug for comments or questions. We will update this bug with the status of our AIs by March 1st, 2024 at the latest, and kindly request that the next-update flag be set to that date.

Comment 5

[Google Trust Services]

Following publication of the report, GTS implemented immediate process changes to mitigate the issue leading to this data preservation gap while we worked to move the remaining few manual validations to an automated solution. The process mitigations provided a safety net while we landed the full automation, which has now been completed. Below is a table that summarizes the specific AIs and the actions taken to address them.

Action Item	Kind	Due Date	Delivery	Steps Taken
Update validation flat files MPA settings to require 2 CA Engineers (CAEs)	Mitigate	2024-01-26	2024-01-26	MPA settings were updated to require approval from 2 CAEs.
Ensure no other IP flat file validations are missing audit evidence	Detect	2024-01-29	2024-01-29	Historical IP address additions to the flat file were reviewed and no other IP address is missing audit evidence.
Update the validation procedures to emphasize that CAEs must record and confirm validation evidence	Mitigate	2024-01-29	2024-01-29	The IP validation procedure was amended with callouts reminding CAEs to record evidence when performing the validation.
Require CA Engineers to confirm they followed the procedure when updating validation flat files	Mitigate	2024-01-29	2024-01-29	An automated change submission check now requires asserting that the validation procedure was followed by including a tag in the change description.
Ensure all validations use GTS’ domain validation service	Prevent	2024-03-01	2024-02-28	Both IP address and domain additions to validation flat files are handled by the same tooling, which relies on GTS' domain validation service described in Bug #1873739.
Automate the proposal of updates to the IPs validation flat file	Prevent	2024-03-01	2024-02-28	Addition of new entries to the validation flat files are automatically proposed by the tooling once validations succeed; 2 CAE approvals are still required to submit the change.
Automate the archival of validation evidence	Prevent	2024-03-01	2024-02-28	GTS' domain validation service automatically stores the validation evidence in a tamper evident log.
Team review of all historical projects & processes	Detect	2024-03-01	2024-02-27	The software engineering team, CA engineers, and technical program managers performed a review of all historical projects and processes. We identified other manual processes unrelated to this incident that we plan to increase automation for. These items have been prioritized for our roadmap. Our review identified an additional action item to update our manual revocation process to use this new automation; an additional action item is described below.
Set permissions of the validation records to only be writable by the role running the automated process	Prevent	2024-03-08	In Progress	The tooling is being extended to support removal of entries before fully locking down permissions. We do not anticipate any issues with completing this before the due date.
Design & implement continual compliance for validation flat files	Detect	2024-05-01	In Progress	The intended design is under review. Since all validation evidence is logged uniformly in our domain validation service, we will be able to extend our existing automated ACME audit automation described in comment#3 to include the logged validation events against the validation flat files.

Based on our review of historical projects and processes, we are adding an additional action item:

Action Item	Kind	Due Date
Update our manual revocation process to use our revised validation automation	Mitigate	2024-03-15

GTS will have an update by March 8th for our next AI. We kindly request that the NextUpdate field be set to then.

Comment 6

[Google Trust Services]

GTS has completed the remediation of the AI due for 2024-03-08 and is continuing work on the last two remaining AIs. All AIs related to this incident, and their status, are listed below.

Action Item	Kind	Due Date	Delivery	Steps Taken
Update validation flat files MPA settings to require 2 CA Engineers (CAEs)	Mitigate	2024-01-26	2024-01-26	Completed, see comment#5
Ensure no other IP flat file validations are missing audit evidence	Detect	2024-01-29	2024-01-29	Completed, see comment#5
Update the validation procedures to emphasize that CAEs must record and confirm validation evidence	Mitigate	2024-01-29	2024-01-29	Completed, see comment#5
Require CA Engineers to confirm they followed the procedure when updating validation flat files	Mitigate	2024-01-29	2024-01-29	Completed, see comment#5
Ensure all validations use GTS’ domain validation service	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Automate the proposal of updates to the IPs validation flat file	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Automate the archival of validation evidence	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Team review of all historical projects & processes	Detect	2024-03-01	2024-02-27	Completed, see comment#5
Set permissions of the validation records to only be writable by the role running the automated process	Prevent	2024-03-08	2024-03-05	The tooling has been extended to support removal of entries and permissions of the validation records have been updated to require updates to be proposed by the role running the automated process; 2 CAE approvals are still required to submit the change.
Update our manual revocation process to use our revised validation automation	Mitigate	2024-03-15	In Progress	The required changes were identified and implementation is underway. We do not anticipate any issues completing this before the due date.
Design & implement continual compliance for validation flat files	Detect	2024-05-01	In Progress	The design is mostly finalized and implementation is underway.

GTS will have an update by March 15th for our next AI. We kindly request that the NextUpdate field be set to that date.

Comment 7

[Google Trust Services]

GTS has completed our action item to update our manual revocation process to use our revised validation automation, which was due on 2024-03-15. We are continuing work on the last remaining AI. All AIs related to this incident, and their status, are listed below.

Action Item	Kind	Due Date	Delivery	Steps Taken
Update validation flat files MPA settings to require 2 CA Engineers (CAEs)	Mitigate	2024-01-26	2024-01-26	Completed, see comment#5
Ensure no other IP flat file validations are missing audit evidence	Detect	2024-01-29	2024-01-29	Completed, see comment#5
Update the validation procedures to emphasize that CAEs must record and confirm validation evidence	Mitigate	2024-01-29	2024-01-29	Completed, see comment#5
Require CA Engineers to confirm they followed the procedure when updating validation flat files	Mitigate	2024-01-29	2024-01-29	Completed, see comment#5
Ensure all validations use GTS’ domain validation service	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Automate the proposal of updates to the IPs validation flat file	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Automate the archival of validation evidence	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Team review of all historical projects & processes	Detect	2024-03-01	2024-02-27	Completed, see comment#5
Set permissions of the validation records to only be writable by the role running the automated process	Prevent	2024-03-08	2024-03-05	Completed, see comment#6
Update our manual revocation process to use our revised validation automation	Mitigate	2024-03-15	2024-03-13	The tooling has been extended to support the validation of revocation requests and procedures have been updated to use it.
Design & implement continual compliance for validation flat files	Detect	2024-05-01	In Progress	The design is finalized and implementation is underway.

GTS will provide another status update regarding our last remaining AI by Friday, April 5. We kindly request that the NextUpdate field be set to that date.

Comment 8

[Google Trust Services]

GTS has completed the remaining action item to implement continual compliance monitoring for validation flat files. The continual compliance monitoring passed QA testing and was enabled in production on 2024-04-04.

All (now resolved) AIs related to this incident are listed below.

Action Item	Kind	Due Date	Delivery	Steps Taken
Update validation flat files MPA settings to require 2 CA Engineers (CAEs)	Mitigate	2024-01-26	2024-01-26	Completed, see comment#5
Ensure no other IP flat file validations are missing audit evidence	Detect	2024-01-29	2024-01-29	Completed, see comment#5
Update the validation procedures to emphasize that CAEs must record and confirm validation evidence	Mitigate	2024-01-29	2024-01-29	Completed, see comment#5
Require CA Engineers to confirm they followed the procedure when updating validation flat files	Mitigate	2024-01-29	2024-01-29	Completed, see comment#5
Ensure all validations use GTS’ domain validation service	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Automate the proposal of updates to the IPs validation flat file	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Automate the archival of validation evidence	Prevent	2024-03-01	2024-02-28	Completed, see comment#5
Team review of all historical projects & processes	Detect	2024-03-01	2024-02-27	Completed, see comment#5
Set permissions of the validation records to only be writable by the role running the automated process	Prevent	2024-03-08	2024-03-05	Completed, see comment#6
Update our manual revocation process to use our revised validation automation	Mitigate	2024-03-15	2024-03-13	Completed, see comment#7
Design & implement continual compliance for validation flat files	Detect	2024-05-01	2024-04-04	Continual compliance monitoring is enabled in the production environment

All AIs were completed within our intended timeline. If there are no further questions or comments, we kindly request to close this incident.

Comment 9

[Google Trust Services]

Google Trust Services continues to monitor this bug for questions or comments. If there are none, we respectfully request that this be closed out.

Comment 10

[Ben Wilson]

I'll close this on Wed. 17-Apr-2024.