1566615 - `addons_daily.addons_daily` job has not run since 2019-07-05

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The last email alert was titled Airflow alert: <TaskInstance: addons_daily.addons_daily 2019-07-05T00:00:00+00:00 [failed]>. The depends_on_past flag is set to True, so subsequent jobs did not run the addons_daily job.

Data has been missing since 2019-07-05.

$ aws s3 ls s3://telemetry-parquet/addons_daily/v1/ | tail -n1 
                           PRE submission_date_s3=20190704/

The expected value should be the current date minus 1 day.

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: GitHub Pull Request

The attached PR should fix the following problem:

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1063 in stage 90.0 failed 4 times, most recent failure: Lost task 1063.3 in stage 90.0 (TID 13185, 10.166.244.8, executor 7): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/worker.py", line 262, in main
    process()
  File "/databricks/spark/python/pyspark/worker.py", line 257, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/databricks/spark/python/pyspark/serializers.py", line 381, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/databricks/spark/python/pyspark/util.py", line 55, in wrapper
    return f(*args, **kwargs)
  File "/databricks/python/lib/python3.5/site-packages/addons_daily/utils/helpers.py", line 75, in <lambda>
    return rp.map(lambda x: x["payload"]["keyedHistograms"]).cache()
KeyError: 'payload'

See: https://workflow.telemetry.mozilla.org/log?task_id=addons_daily&dag_id=addons_daily&execution_date=2019-07-05T00%3A00%3A00%2B00%3A00

Comment 2

[Ben Miroglio [:bmiroglio]]

Hey Anthony--just checking in on the status of this.

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

This is blocked on the attached PR, addons_daily#68.