1312435 - Spark Scheduled Jobs Failing

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Frank Bertsch [:frank]]

Spark scheduled jobs are failing with:
[NbConvertApp] Writing 10503 bytes to ../getting-crash-stats-for-OOM-data-to-S3.ipynb
grep: getting-crash-stats-for-OOM-data-to-S3.ipynb: No such file or directory
Command exiting with ret '0'

It was initially confirmed this was an IOError due to disk space, but that should have been fixed in bug 1311708.

Comment 1

[Frank Bertsch [:frank]]

Discovered that --output in nbconvert just changes the name of the file, am adding --output-dir to put in the current directory.

Comment 2

[Frank Bertsch [:frank]]

mdoglio noted that the airflow.sh will need to be changed as well:

https://github.com/mozilla/telemetry-airflow/blob/master/ansible/files/spark/airflow.sh#L90

Comment 3

[Frank Bertsch [:frank]]

Just tested this using the previous version (Anaconda 4.0, which used nbconvert 4.1.0), and the old code did indeed work as expected. Looks like this was a non-backwards compatible upgrade.

Comment 4

[GitHub Autolander Bot]

Attachment: [emr-bootstrap-spark] fbertsch:hotfix > mozilla:master

Comment 5

[GitHub Autolander Bot]

Attachment: [telemetry-airflow] fbertsch:hotfix > mozilla:master

Comment 6

[Frank Bertsch [:frank]]

Looks like this fixed the issue.

Comment 7

[Mauro Doglio [:mdoglio]]

Thanks a lot :frank