1809127 - Airflow task glam-dev .scalar_percentiles failed on 2022-12-08 & 2022-12-13

Status: RESOLVED INVALID

(Data Platform and Tools :: General, defect, P4)

Description

[Krzysztof Ignasiak [:kik]]

Airflow task glam-dev .scalar_percentiles* failed to complete on 2022-12-13

It appears glam-dev DAG has been stuck since 2022-12-13 this appears to be due to scalar_percentiles being marked as skipped and resulting in the DAG never being marked as successful. What is unclear is why scalar_percentiles was marked as skipped since the previous task according to the logs has exited with status code 0 (success) and I was unable to locate anything in the code indicating this to be expected behaviour.

Comment 1

[Krzysztof Ignasiak [:kik]]

After attempting to re-run the task, we see the following error:

[2023-01-09, 11:05:15 UTC] {{pod_manager.py:226}} INFO - Found versions: []
[2023-01-09, 11:05:15 UTC] {{pod_manager.py:226}} INFO - Source table empty

Which seems to come from here:
https://github.com/mozilla/bigquery-etl/blob/589e9192e7a98b59dcdb814a457575e24b5bec4b/script/glam/run_scalar_agg_clustered_query.py#L50-L52

This results in status code 1 being returned and the pod exits as failed. This status is propagated back to Airflow and therefore the job fails.

We see the same behaviour for run date: 2022-12-08.

In the meantime, marking the tasks in question as failed manually to verify that this fixes the scheduling issues for newer runs.

Comment 2

[Krzysztof Ignasiak [:kik]]

Oops, just noticed this DAG has no schedule and has to be run manually explaining why no recent runs exist. Correcting the bug title to reflect the issue more accurately.

Comment 3

[Krzysztof Ignasiak [:kik]]

Marked as low priority as this does not appear to be a production DAG and is only run on-demand. I assume it is used for testing changes only.

Comment 4

[Krzysztof Ignasiak [:kik]]

:efilho, do we care about these failures? And since this is dev do you think we should just delete this bug ticket?

Comment 5

[Krzysztof Ignasiak [:kik]]

Eduardo confirmed this is fine. Marking this as INVALID and resolved.

Comment 6

[Eduardo Filho [:efilho]]

thank you!