1836271 - Backfill desktop.active_users_aggregates 2018, 2019 and 2020

Status: RESOLVED WORKSFORME

(Data Platform and Tools :: General, enhancement)

Description

[Lucia Vargas [:lucia-vargas-a]]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0

Steps to reproduce:

Data in desktop.active_users_aggregates is available from 2021-01-01.

Actual results:

There is a request to make data available from 2018 for Desktop:
https://mozilla-hub.atlassian.net/browse/DENG-971

Expected results:

Data should be available from 2018-01-01 for Firefox Desktop in the telemetry.active_users_aggregates.

Comment 1

[Lucia Vargas [:lucia-vargas-a]]

Step 1. Generate full snapshot of firefox_desktop_derived.active_users_aggregates_v1 with expiration in 30 days.

Step 2. Backfill moz-fx-data-shared-prod.firefox_desktop_derived.active_users_aggregates_v1 per year. Use --dry-run first.

• Dry-run commands
  bqetl query backfill —dry-run firefox_desktop_derived.active_users_aggregates_v1 --project_id=moz-fx-data-shared-prod -s 2018-01-01 -e 2018-12-31 -n 0
  bqetl query backfill —dry-run firefox_desktop_derived.active_users_aggregates_v1 --project_id=moz-fx-data-shared-prod -s 2019-01-01 -e 2019-12-31 -n 0
  bqetl query backfill —dry-run firefox_desktop_derived.active_users_aggregates_v1 --project_id=moz-fx-data-shared-prod -s 2020-01-01 -e 2020-12-31 -n 0

• Backfill commands
  bqetl query backfill firefox_desktop_derived.active_users_aggregates_v1 --project_id=moz-fx-data-shared-prod -s 2018-01-01 -e 2018-12-31 -n 0
  bqetl query backfill firefox_desktop_derived.active_users_aggregates_v1 --project_id=moz-fx-data-shared-prod -s 2019-01-01 -e 2019-12-31 -n 0
  bqetl query backfill firefox_desktop_derived.active_users_aggregates_v1 --project_id=moz-fx-data-shared-prod -s 2020-01-01 -e 2020-12-31 -n 0

Comment 2

[Lucia Vargas [:lucia-vargas-a]]

Locally, generate the query before running the dry-run and backfill commands:
./script/bqetl generate active_users

Comment 3

[Lucia Vargas [:lucia-vargas-a]]

Backfill completed.