1889144 - Propagate Accounts deletion requests to telemetry

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task)

Description

[Arkadiusz Komarzewski [:akomar]]

We need to propagate Accounts deletion requests to properly shred new telemetry ping added in https://bugzilla.mozilla.org/show_bug.cgi?id=1889123

We should make sure that all telemetry deletion request scenarios, including account deletion, are captured.

Comment 1

[Arkadiusz Komarzewski [:akomar]]

We are currently ingesting Accounts (FxA) deletion requests through legacy logs: https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v2/query.sql that are later used by shredder: https://github.com/mozilla/bigquery-etl/blob/9dbc7cd08bb5da53311f01527a906013f86d16c3/bigquery_etl/shredder/config.py#L108

In order to propagate these deletion requests to Telemetry data, we will:

1. Add raw, unhashed uid to https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v2/query.sql (acceptable per https://docs.google.com/document/d/17eoA4lTdJO6IPjdkzRPT6e7uDbTWM8uGKprgVZA93V8/edit)
2. Build derived, synthetic deletion request table for legacy Telemetry data in telemetry_derived by joining above uid with legacy_client_id sent in https://bugzilla.mozilla.org/show_bug.cgi?id=1889123
3. Create a view to union synthetic legacy Telemetry deletion table with telemetry_stable.deletion_request_v4, and use it instead of this table in Shredder config
4. Build derived, synthetic deletion request table for desktop Glean telemetry in firefox_desktop_derived by joining above uid with client_info.client_id sent in https://bugzilla.mozilla.org/show_bug.cgi?id=1889123, and make Shredder use it by adding it in the config

Comment 2

[Arkadiusz Komarzewski [:akomar]]

In context of https://bugzilla.mozilla.org/show_bug.cgi?id=1895503 :mreid pointed out to me that we want to export just the data from newly added fx_accounts ping table. With that and thinking about the deletion part more I think the approach that the plan above would implement is too broad.

I think accounts deletion request in its current form should not trigger deletion of all the Firefox telemetry by default. To do that, there should either be a separate request, or a flag in the existing request.

Taking above into account, to let us collect client association ping (https://bugzilla.mozilla.org/show_bug.cgi?id=1903310) we will just shred the newly added ping. This means the plan is to:

1. Add raw, unhashed uid to https://github.com/mozilla/bigquery-etl/blob/main/sql/moz-fx-data-shared-prod/firefox_accounts_derived/fxa_delete_events_v2/query.sql (acceptable per https://docs.google.com/document/d/17eoA4lTdJO6IPjdkzRPT6e7uDbTWM8uGKprgVZA93V8/edit)
2. Add new DeleteSource pointing to fxa_delete_events_v2.uid and DeleteTarget pointing to firefox_desktop_stable.fx_accounts_v1 in Shredder config

/cc :mreid for visibility

Comment 3

[BMO Github Automation]

Attachment: [mozilla/bigquery-etl] Bug 1889144 - Add unhashed user id to fxa_delete_events (#5864)

Comment 4

[BMO Github Automation]

Attachment: [mozilla/bigquery-etl] Bug 1889144 - Add fx_accounts_v1 to Shredder config (#5865)