Telemetry backfill completed yesterday and after some parameter tuning structured backfill is expected to complete later tonight. :klukas did some validation work on telemetry and verified that apart from some [some pings from CN being unexpectedly unaffected](https://github.com/mozilla/bigquery-backfill/pull/15/files#diff-558a73df6bb3db6b11a09f94431aaa8696152190ec76fc9b819977528cfab7b8R200), geoip information in the backfilled tables look correct. Assuming structured completes tonight and looks good, we're planning the following for tomorrow: 1. Run the last query to combine [firefox-installer](https://github.com/mozilla/bigquery-backfill/pull/15/commits/65664617915a6910bb6387cc4452df5211a9c230) since this namespace is handled by both the `stub_installer` and `structured` pipeline families 2. run the `_live` partition replacements for `telemetry`, `structured` and `firefox-installer` (special case using the `_combined` backfill table) into shared-prod 3. re-run copy-dedup etc. from airflow for the 5 mainly affected days 4. manually run stub installer copy deduplicate for the 08-17 to 08-29 window (unknown if there are downstream jobs that will need to be rerun) 5. merge https://github.com/mozilla/gcp-ingestion/pull/1815 (note: CI failing) and https://github.com/mozilla-services/cloudops-infra/pull/3340/files, and re-enable schemas and beam deploys, restoring production to fully operational state At a later point when I'm back from PTO: 6. Finalize the tee decommission. We will want to investigate CN specifically since there appears to be a significant DNS propagation delay or similar for that endpoint I also completed the pioneer backfill today, notes in https://github.com/mozilla-services/cloudops-infra/pull/3343.
Bug 1729069 Comment 15 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Telemetry backfill completed yesterday and after some parameter tuning structured backfill is expected to complete later tonight. :klukas did some validation work on telemetry and verified that apart from some [some pings from CN being unexpectedly unaffected](https://github.com/mozilla/bigquery-backfill/pull/15/files#diff-558a73df6bb3db6b11a09f94431aaa8696152190ec76fc9b819977528cfab7b8R200), geoip information in the backfilled tables look correct. Assuming structured completes tonight and looks good, we're planning the following for tomorrow: 1. ~~Run the last query to combine [firefox-installer](https://github.com/mozilla/bigquery-backfill/pull/15/commits/65664617915a6910bb6387cc4452df5211a9c230) since this namespace is handled by both the `stub_installer` and `structured` pipeline families~~ EDIT: done 2. run the `_live` partition replacements for `telemetry`, `structured` and `firefox-installer` (special case using the `_combined` backfill table) into shared-prod 3. re-run copy-dedup etc. from airflow for the 5 mainly affected days 4. manually run stub installer copy deduplicate for the 08-17 to 08-29 window (unknown if there are downstream jobs that will need to be rerun) 5. merge https://github.com/mozilla/gcp-ingestion/pull/1815 (note: CI failing) and https://github.com/mozilla-services/cloudops-infra/pull/3340/files, and re-enable schemas and beam deploys, restoring production to fully operational state At a later point when I'm back from PTO: 6. Finalize the tee decommission. We will want to investigate CN specifically since there appears to be a significant DNS propagation delay or similar for that endpoint I also completed the pioneer backfill today, notes in https://github.com/mozilla-services/cloudops-infra/pull/3343.