1853776 - Airflow task bqetl_search_dashboard.search_derived__search_revenue_levers_daily__v1 failed for exec_date 2023-09-18

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Mike Williams [:mwilliams]]

Airflow task bqetl_search_dashboard.search_derived__search_revenue_levers_daily__v1 failed for exec_date 2023-09-18

Task link:
https://workflow.telemetry.mozilla.org/dags/bqetl_search_dashboard/grid?dag_run_id=scheduled__2023-09-17T04%3A00%3A00%2B00%3A00&task_id=search_derived__search_revenue_levers_daily__v1&tab=logs

Log extract:

[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO - BigQuery error in query operation: Error processing job 'moz-fx-data-shared-
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO - prod:bqjob_r5ed8f685fc4e050a_0000018aa9e46010_1': Invalid schema update. Cannot
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO - add fields (field: dau_w_engine_as_default)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO - Traceback (most recent call last):
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "<string>", line 1, in <module>
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/app/bigquery_etl/cli/__init__.py", line 76, in cli
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     group(prog_name=prog_name)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     return self.main(*args, **kwargs)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     rv = self.invoke(ctx)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     return __callback(*args, **kwargs)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     return f(get_current_context(), *args, **kwargs)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/app/bigquery_etl/cli/query.py", line 876, in run
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     _run_query(
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/app/bigquery_etl/cli/query.py", line 980, in _run_query
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     subprocess.check_call(["bq"] + query_arguments, stdin=query_stream)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO -     raise CalledProcessError(retcode, cmd)
[2023-09-18, 20:03:53 UTC] {pod_manager.py:235} INFO - subprocess.CalledProcessError: Command '['bq', 'query', '--parameter=submission_date:DATE:2023-09-16', '--dataset_id=search_derived', '--project_id=moz-fx-data-shared-prod', '--destination_table=search_revenue_levers_daily_v1$20230916', '--use_legacy_sql=False']' returned non-zero exit status 1.
[2023-09-18, 20:03:54 UTC] {pod_manager.py:260} WARNING - Pod search-derived--search-revenue-levers-daily--v1-vv8w5c5s log read interrupted but container base still running
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO - 
Waiting on bqjob_r5ed8f685fc4e050a_0000018aa9e46010_1 ... (0s) Current status: RUNNING
                                                                                      
Waiting on bqjob_r5ed8f685fc4e050a_0000018aa9e46010_1 ... (0s) Current status: DONE
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO - BigQuery error in query operation: Error processing job 'moz-fx-data-shared-
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO - prod:bqjob_r5ed8f685fc4e050a_0000018aa9e46010_1': Invalid schema update. Cannot
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO - add fields (field: dau_w_engine_as_default)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO - Traceback (most recent call last):
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "<string>", line 1, in <module>
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/app/bigquery_etl/cli/__init__.py", line 76, in cli
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     group(prog_name=prog_name)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     return self.main(*args, **kwargs)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     rv = self.invoke(ctx)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     return __callback(*args, **kwargs)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     return f(get_current_context(), *args, **kwargs)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/app/bigquery_etl/cli/query.py", line 876, in run
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     _run_query(
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/app/bigquery_etl/cli/query.py", line 980, in _run_query
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     subprocess.check_call(["bq"] + query_arguments, stdin=query_stream)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -   File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO -     raise CalledProcessError(retcode, cmd)
[2023-09-18, 20:03:55 UTC] {pod_manager.py:235} INFO - subprocess.CalledProcessError: Command '['bq', 'query', '--parameter=submission_date:DATE:2023-09-16', '--dataset_id=search_derived', '--project_id=moz-fx-data-shared-prod', '--destination_table=search_revenue_levers_daily_v1$20230916', '--use_legacy_sql=False']' returned non-zero exit status 1.
[2023-09-18, 20:03:55 UTC] {pod_manager.py:288} INFO - Pod search-derived--search-revenue-levers-daily--v1-vv8w5c5s has phase Running
[2023-09-18, 20:03:57 UTC] {kubernetes_pod.py:691} INFO - Skipping deleting pod: search-derived--search-revenue-levers-daily--v1-vv8w5c5s
[2023-09-18, 20:03:57 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/google/cloud/operators/kubernetes_engine.py", line 532, in execute
    result = super().execute(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 516, in execute
    return self.execute_sync(context)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 545, in execute_sync
    self.cleanup(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 671, in cleanup
    raise AirflowException(
airflow.exceptions.AirflowException: Pod search-derived--search-revenue-levers-daily--v1-vv8w5c5s returned a failure:

There's a huge JSON object after this.

Comment 1

[Arkadiusz Komarzewski [:akomar]]

Restarting fixed this.