1915996 - Airflow task socorro_import.bigquery_load failed for exec_date 2024-08-30

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[anicholson]

Airflow task socorro_import.bigquery_load failed for exec_date 2024-08-30

Task link:
https://workflow.telemetry.mozilla.org/dags/socorro_import/grid?dag_run_id=scheduled__2024-08-29T00%3A00%3A00%2B00%3A00&task_id=bigquery_load&tab=logs

Log extract:

airflow.providers.cncf.kubernetes.utils.pod_manager.PodLaunchFailedException: Pod took longer than 120 seconds to start. Check the pod events in kubernetes to determine why.

Comment 1

[Ben Wu [:benwu]]

The pod is failing to pull the docker image (Back-off pulling image "docker.io/mozilla/parquet2bigquery:20190722": ImagePullBackOff in gke) and it seems to be because it's not available in docker hub anymore https://www.docker.com/mozilla/parquet2bigquery. I don't know why though. We could put the image in gcr but the repo is archived and hasn't been updated in 5 years so I don't know if that makes sense to do https://github.com/mozilla/parquet2bigquery

Comment 2

[Wesley Dawson [:whd]]

I deleted this image assuming it wasn't used anymore in https://mozilla-hub.atlassian.net/browse/DSRE-1739?focusedCommentId=927691.

I've attempted to rebuild it based on https://github.com/mozilla/parquet2bigquery/pull/36, which has yielded https://github.com/mozilla/telemetry-airflow/pull/2084.

Comment 3

[Wesley Dawson [:whd]]

https://github.com/mozilla/telemetry-airflow/pull/2084 seems to have fixed it. I cleared the tasks for 2024-08-30 forward.

Comment 4

[Sean Rose [:srose]]

All the previously failed tasks that :whd cleared completed successfully, and new tasks since then have also run successfully.