1591277 - sync_view.sync_view_bigquery_load consistently failing since 2019-09-18

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Anthony Miyaguchi [:amiyaguchi]]

See the WTMO dashboard for the failed jobs:
https://workflow.telemetry.mozilla.org/tree?base_date=2019-10-23+00%3A00%3A00%2B00%3A00&num_runs=50&root=&dag_id=sync_view

The bigquery load job for sync_summary_v2 has been failing since 2019-09-18. A query in STMO using the BigQuery beta data source corroborates the failures in Airflow:

https://sql.telemetry.mozilla.org/queries/65630/source#166736

A listing of the aws bucket shows that the data is being generated correctly:

aws s3 ls s3://telemetry-parquet/sync_summary/v2/ | sort -r | head -n7 
2019-10-23 17:11:57          0 submission_date_s3=20191023_$folder$
2019-10-22 17:11:56          0 submission_date_s3=20191022_$folder$
2019-10-21 17:12:07          0 submission_date_s3=20191021_$folder$
2019-10-20 17:11:31          0 submission_date_s3=20191020_$folder$
2019-10-19 17:11:46          0 submission_date_s3=20191019_$folder$
2019-10-18 17:12:03          0 submission_date_s3=20191018_$folder$
2019-10-17 17:12:50          0 submission_date_s3=20191017_$folder$

The common failure seems to have the following message:

[2019-10-24 22:37:15,741] {logging_mixin.py:95} INFO - [2019-10-24 22:37:15,741] {pod_launcher.py:104} INFO - 2019-10-24 22:37:15,663 root ERROR Process-0: exception tuple index out of range. Notifying main process to handle

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Here is a link to one of the most recent logs: https://workflow.telemetry.mozilla.org/log?task_id=bigquery_load&dag_id=sync_view.sync_view_bigquery_load&execution_date=2019-10-23T00%3A00%3A00%2B00%3A00

@jason Do you know if there have been any breaking changes to the load_to_bigquery subdag function? The load job in the sync_view dag hasn't been modified in the last half year.

Comment 3

[Mark Reid [:mreid]]

I see that the sync_summary dag is turned off, is this bug still needed?

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

No, this bug is no longer relevant.