1902284 - Airflow task bqetl_mobile_feature_usage.fenix_derived__feature_usage_events__v1 and fenix_derived__feature_usage_metrics__v1 failed for exec_date 2024-06-13

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Leli [:Leli]]

Airflow task bqetl_mobile_feature_usage.fenix_derived__feature_usage_events__v1 and fenix_derived__feature_usage_metrics__v1 failed for exec_date 2024-06-13

Task link:
https://workflow.telemetry.mozilla.org/dags/bqetl_mobile_feature_usage/grid?dag_run_id=scheduled__2024-06-12T06%3A00%3A00%2B00%3A00&task_id=fenix_derived__feature_usage_events__v1

Log extract:

 STRING, va...; failed to parse view 'fenix.metrics' at [23:1]
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base] Traceback (most recent call last):
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "<string>", line 1, in <module>
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/app/bigquery_etl/cli/__init__.py", line 76, in cli
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     group(prog_name=prog_name)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     return self.main(*args, **kwargs)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/rich_click/rich_command.py", line 126, in main
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     rv = self.invoke(ctx)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]          ^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     return ctx.invoke(self.callback, **ctx.params)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     return __callback(*args, **kwargs)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     return f(get_current_context(), *args, **kwargs)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/app/bigquery_etl/cli/query.py", line 868, in run
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     _run_query(
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/app/bigquery_etl/cli/query.py", line 1013, in _run_query
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     subprocess.check_call(["bq"] + query_arguments, stdin=query_stream)
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]   File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call
[2024-06-13, 07:03:11 UTC] {pod_manager.py:466} INFO - [base]     raise CalledProcessError(retcode, cmd)
[2024-06-13, 07:03:12 UTC] {pod_manager.py:483} INFO - [base] subprocess.CalledProcessError: Command '['bq', 'query', '--parameter=submission_date:DATE:2024-06-12', '--project_id=moz-fx-data-shared-prod', '--destination_table=feature_usage_events_v1$20240612', '--use_legacy_sql=False', '--dataset_id=fenix_derived']' returned non-zero exit status 1.
[2024-06-13, 07:03:12 UTC] {pod_manager.py:616} INFO - Pod fenix-derived--feature-usage-events--v1-rbkuy2s0 has phase Running
[2024-06-13, 07:03:15 UTC] {pod.py:912} INFO - Skipping deleting pod: fenix-derived--feature-usage-events--v1-rbkuy2s0
[2024-06-13, 07:03:15 UTC] {taskinstance.py:2728} ERROR - Task failed with exception

Comment 1

[Lucia Vargas]

Related slack thread https://mozilla.slack.com/archives/C4D5ZA91B/p1718709126449019

Main error:

Error in query string: Error processing job 'moz-fx-data-shared-prod:bqjob_r6a275a2b4cc04375_000001902a27ed4a_1' 

Failed to parse view 'fenix.metrics' : 

Column 8 in UNION ALL has incompatible types:

STRUCT<boolean STRUCT<gfx_status_headless BOOL,
glean_core_migration_successful BOOL,
glean_error_preinit_tasks_timeout BOOL,
counter STRUCT<events_total_uri_count INT64,
glean_error_preinit_tasks_overflow INT64,
glean_validation_app_forceclosed_count INT64, ...>, 
custom_distribution STRUCT<geckoview_document_site_origins STRUCT<sum INT64, values ARRAY<STRUCT<key STRING, value INT64>>,
count INT64>, 
gfx_checkerboard_peak_pixel_count STRUCT<sum INT64, values ARRAY<STRUCT<key STRING, value INT64>>, count INT64>, 
gfx_checkerboard_severity STRUCT<sum INT64, values ARRAY<STRUCT<key STRING, value INT64>>, count INT64>, ...>, ...>, STRUCT<boolean STRUCT<gfx_status_headless BOOL,
glean_core_migration_successful BOOL, 
glean_error_preinit_tasks_timeout BOOL, 
STRUCT<events_total_uri_count INT64, glean_error_preinit_tasks_overflow INT64, glean_validation_app_forceclosed_count INT64, ...>, custom_distribution STRUCT<geckoview_document_site_origins STRUCT<sum INT64, values ARRAY<STRUCT<key STRING, va...;

Comment 2

[Ben Wu [:benwu]]

This is caused by the issue described in bug 1883727. The views were just redeployed so it should be working now.

Comment 3

[Ben Wu [:benwu]]

I reran the failed tasks from today and last week so the current issue now resolved. The issue may come up again though because of bug 1883727. so it might be worthwhile to document somewhere more visible