1470547 - Airflow alert: <TaskInstance: main_summary.retention 2018-06-21 01:00:00 [failed]>

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Try 3 out of 3
Exception:
Spark job 1-Day Firefox Retention terminated with errors: STEP_FAILURE - Shut down as step failed
Log: Link
Host: ip-172-31-47-40.us-west-2.compute.internal
Log file: /app/logs/main_summary/retention/2018-06-21T01:00:00.log
Mark success: Link

The job fails due to a missing dependency:

```
[0m[[0m[31merror[0m] [0m[0m/mnt/tmp/tmp.bH4RV4xFY8/telemetry-batch-view/src/main/scala/com/mozilla/telemetry/views/GenericCountView.scala:6:19: object nscala_time is not a member of package com.github[0m
[0m[[0m[31merror[0m] [0m[0mimport com.github.nscala_time.time.Imports._[0m
[0m[[0m[31merror[0m] [0m[0m                  ^[0m
[0m[[0m[31merror[0m] [0m[0m/mnt/tmp/tmp.bH4RV4xFY8/telemetry-batch-view/src/main/scala/com/mozilla/telemetry/views/GenericCountView.scala:89:21: not found: value DateTimeFormat[0m
[0m[[0m[31merror[0m] [0m[0m  private val fmt = DateTimeFormat.forPattern("yyyyMMdd")[0m
[0m[[0m[31merror[0m] [0m[0m                    ^[0m
```

The logs are also emitting warning from the scala style enforcement. 

https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-2GJGHKQO9DPX1

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

The retention job is also compiling the packages from source on run, this goes against our current deployment process. This should be moving back to python_mozetl with pyspark-hyperloglog.

https://github.com/mozilla/telemetry-airflow/blob/master/jobs/retention.sh

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Add back nscala-time dependency #444

Thanks to :klukas for explaining the spark 2.3 patches that were recently merged in and :relud for adding the dependency back.

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

I am backfilling both retention and client_count_daily_view since they both failed from the same missing dependency.

Comment 4

[Anthony Miyaguchi [:amiyaguchi]]

> $ aws s3 ls s3://telemetry-parquet/retention/v1/start_date=20180621/ | grep _SUCCESS
> 2018-06-22 14:44:29          0 _SUCCESS

The backfill is complete. The logs can be found in EMR, linked below.

Client Count: https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-15XSA8M3B2LDC
Retention: https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-3ORVLWH4JWBCN