1809623 - Airflow task mobile_aggregates.mobile_aggregate_view_dataproc failed on 2023-01-10

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Krzysztof Ignasiak [:kik]]

Airflow task mobile_aggregates.mobile_aggregate_view_dataproc failed on 2023-01-10

It appears currently our dataproc clusters fail to initialize correctly:

[2023-01-11, 11:02:41 UTC] {{taskinstance.py:1909}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 598, in execute
    cluster = self._create_cluster(hook)
  File "/usr/local/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/dataproc.py", line 533, in _create_cluster
    cluster = operation.result()
  File "/usr/local/lib/python3.8/site-packages/google/api_core/future/polling.py", line 137, in result
    raise self._exception
google.api_core.exceptions.InvalidArgument: 400 Multiple Errors:
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/python/pip-install.sh', see output in: gs://moz-fx-data-prod-dataproc-scratch/google-cloud-dataproc-metainfo/b8a953c3-97d2-4267-a5ad-7d9c438598a0/mobile-metrics-aggregates-20230110-m/dataproc-initialization-script-0_output
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/python/pip-install.sh', see output in: gs://moz-fx-data-prod-dataproc-scratch/google-cloud-dataproc-metainfo/b8a953c3-97d2-4267-a5ad-7d9c438598a0/mobile-metrics-aggregates-20230110-w-0/dataproc-initialization-script-0_output
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/python/pip-install.sh', see output in: gs://moz-fx-data-prod-dataproc-scratch/google-cloud-dataproc-metainfo/b8a953c3-97d2-4267-a5ad-7d9c438598a0/mobile-metrics-aggregates-20230110-w-1/dataproc-initialization-script-0_output
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/python/pip-install.sh', see output in: gs://moz-fx-data-prod-dataproc-scratch/google-cloud-dataproc-metainfo/b8a953c3-97d2-4267-a5ad-7d9c438598a0/mobile-metrics-aggregates-20230110-w-4/dataproc-initialization-script-0_output
[2023-01-11, 11:02:41 UTC] {{taskinstance.py:1415}} INFO - Marking task as FAILED. dag_id=mobile_aggregates.mobile_aggregate_view_dataproc, task_id=create_dataproc_cluster, execution_date=20230110T000000, start_date=20230111T105839, end_date=20230111T110241
[2023-01-11, 11:02:41 UTC] {{warnings.py:109}} WARNING - /usr/local/lib/python3.8/site-packages/airflow/utils/email.py:119: PendingDeprecationWarning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
  send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)

It appears the root cause is the same as for:
https://bugzilla.mozilla.org/show_bug.cgi?id=1809616

Comment 1

[Krzysztof Ignasiak [:kik]]

The job completed successfully for this data.

Summary:
moztelemetry package was removed from pypi causing the installation of python-mozaggregator package to fail. For now we solved this by setting the dependency to install moztelemtry from its Github repo:
https://github.com/mozilla/python_mozaggregator/pull/235

Long term we should consider deprecating this dependency.