1770823 - Airflow task taar_weekly.dataflow_import_avro_to_bigtable failing on 2022-05-22

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Linh Nguyen [:linh]]

The Airflow task taar_weekly.dataflow_import_avro_to_bigtable failed on 2022-05-22. See error log here.

Comment 1

[Alekhya Kommasani [:Alekhya]]

The Airflow task taar_weekly.dataflow_import_avro_to_bigtable failed on 2022-07-10.
The Airflow task taar_weekly.taar_lite https://workflow.telemetry.mozilla.org/tree?dag_id=taar_daily

The issues seems to be : pyspark.sql.utils.AnalysisException: 'Path does not exist: gs://moz-fx-data-derived-datasets-parquet/clients_daily/v6/submission_date_s3=20220720;'

Comment 2

[Evgeny Pavlov]

Is gs://moz-fx-data-derived-datasets-parquet/clients_daily/v6 still a relevant data source or it was deprecated and we should migrate to something else?

Comment 3

[Daniel Thorn [:relud]]

if you can migrate to reading mozdata.telemetry.clients_daily (or moz-fx-data-shared-prod.telemetry_derived.clients_daily_v6), that would be good.

the parquet export for gs://moz-fx-data-derived-datasets-parquet/clients_daily/v6/submission_date_s3=20220720 seems to have started failing while i was on PTO. it's still maintained exclusively for this use case, and I haven't had a chance to investigate yet.

Comment 4

[Evgeny Pavlov]

:relud is it just replacing the name or switching to reading from BigQuery?

If the latter, it would require changing and retesting three TAAR jobs to query BigQuery instead of Spark and there's nobody to work on that. I'm still the maintainer but I'm at Pocket now, so I can only make quick fixes. Maybe the data engineering has the resources to work on this? Or we can just stick to the old scheme with parquet.

Comment 5

[Daniel Thorn [:relud]]

definitely stick with parquet then

Comment 6

[Daniel Thorn [:relud]]

https://github.com/mozilla/telemetry-airflow/pull/1531 should get parquet working again

Comment 7

[Daniel Thorn [:relud]]

also blocked on https://github.com/mozilla/telemetry-airflow/pull/1532

Comment 8

[Daniel Thorn [:relud]]

parquet exports are fixed now

Comment 9

[Frank Bertsch [:frank]]

Job is now working. Evgeny, do we need to backfill the failed tasks, or can we mark this complete?

Comment 10

[Evgeny Pavlov]

We don't need to backfill, closing