1905989 - Airflow task bqetl_mozilla_vpn_site_metrics.mozilla_vpn_derived__site_metrics_empty_check__v1 failed since exec_dates 2024-06-30

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Winnie Chan [:winnie]]

Airflow task bqetl_mozilla_vpn_site_metrics.mozilla_vpn_derived__site_metrics_empty_check__v1 failed since exec_dates 2024-06-30

https://workflow.telemetry.mozilla.org/dags/bqetl_mozilla_vpn_site_metrics/grid?dag_run_id=scheduled__2024-07-02T15%3A00%3A00%2B00%3A00&task_id=mozilla_vpn_derived__site_metrics_empty_check__v1

[2024-07-02, 00:16:22 UTC] {pod_manager.py:466} INFO - [base] Error in query string: Error processing job 'moz-fx-data-shared-
[2024-07-02, 00:16:22 UTC] {pod_manager.py:466} INFO - [base] prod:bqjob_r1b758ad5532e093a_0000019070ccada3_1': No data for mozilla.org on
[2024-07-02, 00:16:22 UTC] {pod_manager.py:466} INFO - [base] 2024-06-30

Comment 1

[Sean Rose [:srose]]

The mozilla_vpn_derived.site_metrics_empty_check_v1 ETL checks that the GA3 data needed by the mozilla_vpn_derived.site_metrics_summary_v1 ETL exists, and it's failing because Google has now permanently stopped sending GA3 data to BigQuery.

While a new mozilla_vpn_derived.site_metrics_summary_v2 ETL based on GA4 data was added in https://github.com/mozilla/bigquery-etl/pull/5227, the mozilla_vpn_derived.funnel_ga_to_subscriptions_v1 ETL still depends on mozilla_vpn_derived.site_metrics_summary_v1.

My current plan for resolving this involves:

• Changing the mozilla_vpn.site_metrics_summary view to combine the data from mozilla_vpn_derived.site_metrics_summary_v1 and mozilla_vpn_derived.site_metrics_summary_v2.
• Probably removing the mozilla_vpn.site_metrics_summary_v2 view.
• Updating the mozilla_vpn_derived.funnel_ga_to_subscriptions_v1 ETL to use the mozilla_vpn.site_metrics_summary view instead of mozilla_vpn_derived.site_metrics_summary_v1.
• Unscheduling the mozilla_vpn_derived.site_metrics_summary_v1 ETL.
• Removing the mozilla_vpn_derived.site_metrics_empty_check_v1 ETL entirely.

However, that will have to wait until next week as tomorrow is a holiday in the US and Friday is a company-wide day off. For now I'm temporarily pausing the bqetl_mozilla_vpn_site_metrics DAG until I can resolve this next week.

Comment 2

[Matthew Gaudet (he/him) [:mgaudet]]

Attachment: Bug 1905989 - Clear pending invalidation dependencies when a script is invalidated r?iain,sfink!

Comment 3

[Phabricator Automation]

Comment on attachment 9411624 [details]
Bug 1905989 - Clear pending invalidation dependencies when a script is invalidated r?iain,sfink!

Revision D215816 was moved to bug 1905988. Setting attachment 9411624 [details] to obsolete.

Comment 4

[BMO Github Automation]

Attachment: [mozilla/bigquery-etl] Add `mozilla_vpn_derived.funnel_ga_to_subscriptions_v2` ETL using GA4 data (bug 1905989) (#6033)

Comment 5

[Sean Rose [:srose]]

This has been resolved by bigquery-etl PR 6033 "Add mozilla_vpn_derived.funnel_ga_to_subscriptions_v2 ETL using GA4 data".

I've unpaused the bqetl_mozilla_vpn_site_metrics DAG.