Airflow task taar_weekly.dataflow_import_avro_to_bigtable run for 2023-02-05 appears stuck
Airflow logs show that the pod has not been started:
```
[2023-02-12, 00:01:58 UTC] {{kubernetes_pod.py:564}} INFO - Creating pod dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326 with labels: {'dag_id': 'taar_weekly', 'task_id': 'dataflow_import_avro_to_bigtable', 'run_id': 'scheduled__2023-02-05T0000000000-64592e701', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-02-12, 00:02:00 UTC] {{pod_manager.py:178}} WARNING - Pod not yet started: dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326
[2023-02-12, 00:02:01 UTC] {{pod_manager.py:178}} WARNING - Pod not yet started: dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326
[2023-02-12, 00:02:02 UTC] {{pod_manager.py:178}} WARNING - Pod not yet started: dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326
```
When checking for the pod under "workloads" inside GCP we see the container exists, however, it appears the application is stuck and has not produced any logs for more than 24 hours now. Last logs:
```
2023-02-12 01:02:12.935 CET
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
2023-02-12 01:02:14.947 CET
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iso-date=20230205', '--gcp-project=moz-fx-data-taar-pr-prod-e0f7', '--avro-gcs-bucket=moz-fx-data-taar-pr-prod-e0f7-prod-etl', '--bigtable-instance-id=taar-prod-202006', '--sample-rate=1.0', '--dataflow-service-account=taar-prod-dataflow@moz-fx-data-taar-pr-prod-e0f7.iam.gserviceaccount.com', '--gcs-to-bigtable']
2023-02-12 01:02:14.951 CET
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iso-date=20230205', '--gcp-project=moz-fx-data-taar-pr-prod-e0f7', '--avro-gcs-bucket=moz-fx-data-taar-pr-prod-e0f7-prod-etl', '--bigtable-instance-id=taar-prod-202006', '--sample-rate=1.0', '--dataflow-service-account=taar-prod-dataflow@moz-fx-data-taar-pr-prod-e0f7.iam.gserviceaccount.com', '--gcs-to-bigtable']
```
Bug 1816446 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Airflow task taar_weekly.dataflow_import_avro_to_bigtable run for 2023-02-05 appears stuck
Airflow logs show that the pod has not been started:
```
[2023-02-12, 00:01:58 UTC] {{kubernetes_pod.py:564}} INFO - Creating pod dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326 with labels: {'dag_id': 'taar_weekly', 'task_id': 'dataflow_import_avro_to_bigtable', 'run_id': 'scheduled__2023-02-05T0000000000-64592e701', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-02-12, 00:02:00 UTC] {{pod_manager.py:178}} WARNING - Pod not yet started: dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326
[2023-02-12, 00:02:01 UTC] {{pod_manager.py:178}} WARNING - Pod not yet started: dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326
[2023-02-12, 00:02:02 UTC] {{pod_manager.py:178}} WARNING - Pod not yet started: dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326
```
When checking for the pod under "workloads" inside GCP we see the container exists, however, it appears the application is stuck and has not produced any logs for more than 24 hours now. Last logs:
```
2023-02-12 01:02:12.935 CET
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
2023-02-12 01:02:14.947 CET
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iso-date=20230205', '--gcp-project=moz-fx-data-taar-pr-prod-e0f7', '--avro-gcs-bucket=moz-fx-data-taar-pr-prod-e0f7-prod-etl', '--bigtable-instance-id=taar-prod-202006', '--sample-rate=1.0', '--dataflow-service-account=taar-prod-dataflow@moz-fx-data-taar-pr-prod-e0f7.iam.gserviceaccount.com', '--gcs-to-bigtable']
2023-02-12 01:02:14.951 CET
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--iso-date=20230205', '--gcp-project=moz-fx-data-taar-pr-prod-e0f7', '--avro-gcs-bucket=moz-fx-data-taar-pr-prod-e0f7-prod-etl', '--bigtable-instance-id=taar-prod-202006', '--sample-rate=1.0', '--dataflow-service-account=taar-prod-dataflow@moz-fx-data-taar-pr-prod-e0f7.iam.gserviceaccount.com', '--gcs-to-bigtable']
```
https://console.cloud.google.com/kubernetes/pod/us-west1/workloads-prod-v1/default/dataflow-import-avro-to-bigtab-df1e603f15af4a748a2e1f8205a1c326/details?project=moz-fx-data-airflow-gke-prod&pageState=(%22savedViews%22:(%22i%22:%2288c77fdb0e024497ad0c5ff75f746848%22,%22c%22:%5B%22gke%2Fus-west1%2Fworkloads-prod-v1%22%5D,%22n%22:%5B%5D))