1697602 - Remove AET support in the data pipeline

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task, P2)

Description

[Jeff Klukas [:klukas] (UTC-4)]

The Account Ecosystem Telemetry project is no longer being pursued and a decision has been made to remove the components that were built to support it. For the data pipeline, this loosely looks like:

• bigquery-etl: Remove the ETL defined in account_ecosystem* folders and delete the underlying tables
• cloudops-infra: Remove the account_ecosystem* datasets, access groups, Dataflow job configurations, etc.
• gcp-ingestion: Remove the AET-specific decoder logic

I will plan to propose some PRs, and then spin off an ops bug.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/gcp-ingestion/pull/1599

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/gcp-ingestion/pull/1600

Comment 3

[Jeff Klukas [:klukas] (UTC-4)]

https://github.com/mozilla/gcp-ingestion/pull/1600 is now merged. Once that's deployed, it means that we should no longer see any new messages getting populated in the live and stable tables for firefox-accounts/account-ecosystem or telemetry/account-ecosystem.

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/1894

Comment 5

[Wesley Dawson [:whd]]

I deployed https://github.com/mozilla/gcp-ingestion/pull/1600 so UTC the 16th and forward should be empty.

For the ops side bug we're going to need to coordinate with FxA-side efforts WRT the pubsub topic. It may be that we decide to keep the topic in low retention (10m) mode as well as the edge config as the way to discard AET messages, but we certainly don't want to remove the topic until the FxA-side log sink is decommissioned (I get enough [ACTION REQUIRED] Cloud Logging export config error emails as it is).

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/664

Comment 7

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/mozilla-schema-generator/pull/185

Comment 8

[Jeff Klukas [:klukas] (UTC-4)]

The queries and DAG are now removed. I have also gone in by hand and deleted the user-facing view for the derived data along with all the derived tables. I did not touch the account_ecosystem_restricted dataset, as there may be ops logic that controls the existence of the encrypted_keys_v1 table, so I defer that to a stage where Data SRE is removing configuration related to this.

Comment 9

[Wesley Dawson [:whd]]

I did not touch the account_ecosystem_restricted dataset, as there may be ops logic that controls the existence of the encrypted_keys_v1 table

This table is generated manually via a command stored in secrets-sops-default-dataops/misc/aet_hmac.yaml. It's safe to remove since there isn't any automation associated with it. I've gone ahead and deleted that file and secrets-sops-default-dataops/misc/aet.yaml (the SOPS backup of the AET keys) as well.

Comment 10

[Jeff Klukas [:klukas] (UTC-4)]

It's safe to remove since there isn't any automation associated with it

Sounds good. I've now removed both of the tables in account_ecosystem_restricted.

Comment 11

[Jeff Klukas [:klukas] (UTC-4)]

jbuck has removed the log sinks in https://bugzilla.mozilla.org/show_bug.cgi?id=1698798 so the Pub/Sub topic that we use for ingesting FxA server logs should be empty going forward. I believe that means :whd should feel free to remove the topics and the Dataflow jobs related to AET. We can merge the gcp-ingestion changes once we no longer have any Dataflow jobs deployed that depend on the AET-specific Decoder options.

We should also be good to remove the account_ecosystem* datasets.

Comment 12

[Wesley Dawson [:whd]]

(In reply to Jeff Klukas [:klukas] (UTC-4) from comment #11)

I believe that means :whd should feel free to remove the topics and the Dataflow jobs related to AET. We can merge the gcp-ingestion changes once we no longer have any Dataflow jobs deployed that depend on the AET-specific Decoder options.

I have the cleanup branch prepared in https://github.com/mozilla-services/cloudops-infra/compare/data_aet_cleanup?expand=1 but haven't PR'd it since order of operations is important and it's likely easier to handle this with two PRs (one for removing the topic and beam resources and the other for removing BQ resources). The plan looks something like:

1. Redeploy edge routing rules
   This will divert AET data to the structured and telemetry raw topics instead of their corresponding AET topics, which are then dropped by https://github.com/mozilla/gcp-ingestion/pull/1600. I'll throw in the unrelated rally routing change for operational convenience.
2. Destroy terraform, KMS, and beam resources (PR 1)
3. Destroy bigquery resources (PR 2 and https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/664 https://github.com/mozilla/mozilla-schema-generator/pull/185)
4. Merge and deploy final code cleanup PR: https://github.com/mozilla/gcp-ingestion/pull/1599

Comment 13

[Wesley Dawson [:whd]]

I landed https://github.com/mozilla-services/cloudops-infra/pull/2969 today with plans to merge and deploy https://github.com/mozilla-services/cloudops-infra/pull/2970 https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/664 and https://github.com/mozilla/mozilla-schema-generator/pull/185 in concert tomorrow.

I'm treating this as higher priority than it might be considered otherwise because it will simplify the work in bug #1692117.

Comment 14

[Wesley Dawson [:whd]]

I've landed the remaining PRs. There was a minor issue with recent probe-scraper/MSG refactoring causing MSG CI to fail to push to dockerhub which :amiyaguchi helped me resolve in https://github.com/mozilla/mozilla-schema-generator/pull/186, and mozdata-nonprod failed to destroy the AET on initial deploy because aet_clients_daily wasn't cleaned up beforehand (not a problem in pro). I landed https://github.com/mozilla/gcp-ingestion/pull/1599 immediately because :relud indicated tests had started failing.