Open Bug 1940789 Opened 1 month ago

Airflow task operational_monitoring.opmon_run failed for exec_date 2025-01-09

Categories

(Data Platform and Tools :: General, defect)

defect

Tracking

(Not tracked)

People

(Reporter: mwilliams, Unassigned)

Details

(Whiteboard: [airflow-triage])

Airflow task operational_monitoring.opmon_run failed for exec_date 2025-01-09

Task link:
https://workflow.telemetry.mozilla.org/dags/operational_monitoring/grid?dag_run_id=scheduled__2025-01-08T04%3A00%3A00%2B00%3A00&task_id=opmon_run&tab=logs

Log extract:

[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] Traceback (most recent call last):
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/bin/opmon", line 8, in <module>
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     sys.exit(cli())
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return self.main(*args, **kwargs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     rv = self.invoke(ctx)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return _process_result(sub_ctx.command.invoke(sub_ctx))
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return ctx.invoke(self.callback, **ctx.params)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return __callback(*args, **kwargs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/opmon/cli.py", line 269, in run
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     results = pool.map(run, configs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 367, in map
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return self._map_async(func, iterable, mapstar, chunksize).get()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     raise self._value
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     result = (True, func(*args, **kwds))
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return list(map(*args))
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/opmon/cli.py", line 295, in _run
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     monitoring.run(submission_date)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/opmon/monitoring.py", line 84, in run
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     self.bigquery.execute(
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/opmon/bigquery_client.py", line 142, in execute
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     job.result()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/job/query.py", line 1499, in result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     do_get_result()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return retry_target(
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return target()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/job/query.py", line 1489, in do_get_result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     super(QueryJob, self).result(retry=retry, timeout=timeout)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/job/base.py", line 728, in result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]   File "/usr/local/lib/python3.10/site-packages/google/api_core/future/polling.py", line 261, in result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base]     raise self._exception
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] google.api_core.exceptions.NotFound: 404 Not found: Table moz-fx-data-shared-prod:operational_monitoring_derived.fox_doodle_and_tail_fox_2025_rollout_v1 was not found in location US
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] Location: US
[2025-01-09, 06:44:14 UTC] {pod_manager.py:490} INFO - [base] Job ID: 9cd8a853-0b43-45cb-93bd-b9c6e8b8bd53
You need to log in before you can comment on or make changes to this bug.