1838416 - Airflow task jetstream. jetstream_run_config_changed failed for exec_date 2023-06-12, 04:00:00

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Leli [:Leli]]

Airflow task jetstream. jetstream_run_config_changed failed for exec_date 2023-06-12, 04:00:00

Task link:
https://prod.telemetry-airflow.prod.dataservices.mozgcp.net/dags/jetstream/grid?root=&dag_run_id=scheduled__2023-06-12T04%3A00%3A00%2B00%3A00&task_id=jetstream_run_config_changed

Log extract:

*** Unable to read remote log from gs://airflow-remote-logs-prod-prod/dag_id=jetstream/run_id=scheduled__2023-06-12T04:00:00+00:00/task_id=jetstream_run_config_changed/attempt=2.log
*** 404 GET https://storage.googleapis.com/download/storage/v1/b/airflow-remote-logs-prod-prod/o/dag_id%3Djetstream%2Frun_id%3Dscheduled__2023-06-12T04%3A00%3A00%2B00%3A00%2Ftask_id%3Djetstream_run_config_changed%2Fattempt%3D2.log?alt=media: No such object: airflow-remote-logs-prod-prod/dag_id=jetstream/run_id=scheduled__2023-06-12T04:00:00+00:00/task_id=jetstream_run_config_changed/attempt=2.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

*** Log file does not exist: /opt/airflow/logs/dag_id=jetstream/run_id=scheduled__2023-06-12T04:00:00+00:00/task_id=jetstream_run_config_changed/attempt=2.log
*** Fetching from: http://gha-worker-fd498b4f7-ll5wh:8793/log/dag_id=jetstream/run_id=scheduled__2023-06-12T04:00:00+00:00/task_id=jetstream_run_config_changed/attempt=2.log
*** Failed to fetch log file from worker. [Errno -2] Name or service not known

There was a corresponding pod inside the k8s workloads "Running" for prior ~40 minutes, however, when checking the pod logs there was no application logs showing.
We tried to restart the task by clearing it in Airflow and got the same error message.
It is unclear if this is an issue with the application or with the infrastructure. We should investigate and understand what is happening.

Comment 1

[Leli [:Leli]]

cc mducharme in case this is connected to the migration

Comment 2

[Mikaël Ducharme [:mducharme]]

Logs only appear once the task is finished. This is a limitation that I'm looking to fix early next quarter.