Open
Bug 1940789
Opened 1 month ago
Airflow task operational_monitoring.opmon_run failed for exec_date 2025-01-09
Categories
(Data Platform and Tools :: General, defect)
Data Platform and Tools
General
Tracking
(Not tracked)
NEW
People
(Reporter: mwilliams, Unassigned)
Details
(Whiteboard: [airflow-triage])
Airflow task operational_monitoring.opmon_run failed for exec_date 2025-01-09
Log extract:
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] Traceback (most recent call last):
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/bin/opmon", line 8, in <module>
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] sys.exit(cli())
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return self.main(*args, **kwargs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] rv = self.invoke(ctx)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return _process_result(sub_ctx.command.invoke(sub_ctx))
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return ctx.invoke(self.callback, **ctx.params)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return __callback(*args, **kwargs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/opmon/cli.py", line 269, in run
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] results = pool.map(run, configs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 367, in map
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return self._map_async(func, iterable, mapstar, chunksize).get()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] raise self._value
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] result = (True, func(*args, **kwds))
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return list(map(*args))
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/opmon/cli.py", line 295, in _run
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] monitoring.run(submission_date)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/opmon/monitoring.py", line 84, in run
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] self.bigquery.execute(
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/opmon/bigquery_client.py", line 142, in execute
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] job.result()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/job/query.py", line 1499, in result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] do_get_result()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return retry_target(
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return target()
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/job/query.py", line 1489, in do_get_result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] super(QueryJob, self).result(retry=retry, timeout=timeout)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/google/cloud/bigquery/job/base.py", line 728, in result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] File "/usr/local/lib/python3.10/site-packages/google/api_core/future/polling.py", line 261, in result
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] raise self._exception
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] google.api_core.exceptions.NotFound: 404 Not found: Table moz-fx-data-shared-prod:operational_monitoring_derived.fox_doodle_and_tail_fox_2025_rollout_v1 was not found in location US
[2025-01-09, 06:44:13 UTC] {pod_manager.py:472} INFO - [base] Location: US
[2025-01-09, 06:44:14 UTC] {pod_manager.py:490} INFO - [base] Job ID: 9cd8a853-0b43-45cb-93bd-b9c6e8b8bd53
You need to log in
before you can comment on or make changes to this bug.
Description
•