1867535 - Airflow task bqetl_backfill.bqetl_backfill failed for exec_date 2023-11-29

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, defect)

Description

[Mike Williams [:mwilliams]]

Airflow task bqetl_backfill.bqetl_backfill failed for exec_date 2023-11-29

Run:
Backfill+gclid_conversions_v1

Task link:
https://workflow.telemetry.mozilla.org/dags/bqetl_backfill/grid?dag_run_id=Backfill%2Bgclid_conversions_v1&task_id=bqetl_backfill&tab=logs

Log extract:

[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] multiprocessing.pool.RemoteTraceback:
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] """
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] Traceback (most recent call last):
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     result = (True, func(*args, **kwds))
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return list(map(*args))
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/bigquery_etl/cli/query.py", line 550, in _backfill_query
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     _run_query(
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/bigquery_etl/cli/query.py", line 991, in _run_query
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     subprocess.check_call(["bq"] + query_arguments, stdin=query_stream)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     raise CalledProcessError(retcode, cmd)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] subprocess.CalledProcessError: Command '['bq', 'query', '--parameter=activity_date:DATE:2023-08-15', '--use_legacy_sql=false', '--replace', '--max_rows=10', '--project_id=moz-fx-data-shared-prod', '--format=none', '--dry_run', '--dataset_id=mozilla_org_derived', '--project_id=moz-fx-data-shared-prod', '--destination_table=moz-fx-data-shared-prod:mozilla_org_derived.gclid_conversions_v1$20230815', '--use_legacy_sql=False']' returned non-zero exit status 1.
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] """
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] The above exception was the direct cause of the following exception:
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base] Traceback (most recent call last):
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/bin/bqetl", line 8, in <module>
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     sys.exit(cli())
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/bigquery_etl/cli/__init__.py", line 76, in cli
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     group(prog_name=prog_name)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return self.main(*args, **kwargs)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     rv = self.invoke(ctx)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return ctx.invoke(self.callback, **ctx.params)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return __callback(*args, **kwargs)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return f(get_current_context(), *args, **kwargs)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/bigquery_etl/cli/query.py", line 790, in backfill
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     result = p.map(backfill_query, dates, chunksize=1)
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 367, in map
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     return self._map_async(func, iterable, mapstar, chunksize).get()
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get
[2023-11-29, 17:27:10 UTC] {pod_manager.py:437} INFO - [base]     raise self._value
[2023-11-29, 17:27:11 UTC] {pod_manager.py:450} INFO - [base] subprocess.CalledProcessError: Command '['bq', 'query', '--parameter=activity_date:DATE:2023-08-15', '--use_legacy_sql=false', '--replace', '--max_rows=10', '--project_id=moz-fx-data-shared-prod', '--format=none', '--dry_run', '--dataset_id=mozilla_org_derived', '--project_id=moz-fx-data-shared-prod', '--destination_table=moz-fx-data-shared-prod:mozilla_org_derived.gclid_conversions_v1$20230815', '--use_legacy_sql=False']' returned non-zero exit status 1.
[2023-11-29, 17:27:11 UTC] {pod_manager.py:582} INFO - Pod bqetl-backfill-urmnuu2n has phase Running