848680 - add pig support to crontabber

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[Robert Helmer [:rhelmer]]

crontabber has classes to help with calling postgres stored procedures (PostgresCronApp and friends), we should have something similar for Pig jobs.

This would make bug 847735 more palatable, and easier for us to add new jobs.

There's only one pig job right now (https://github.com/mozilla/socorro/blob/master/scripts/crons/cron_modulelist.sh) which needs to call out to the external "pig" executable, and do a couple followup tasks like copy out the report, cleanup temporary files etc.

Making it easy to load the results directly into postgres would be nice to have too, since that's something we have been talked about doing.

Here is an example of what we're doing now:
https://github.com/mozilla/socorro/blob/master/scripts/crons/cron_modulelist.sh

Comment 1

[Peter Bengtsson [:peterbe]]

I think the right thing to do would be to base it on BaseBackfillCronApp [1] and simply execute the job with subprocess. 

I'm pretty poor when it comes to pig, hadoop and stuff with HBase but I can write the job as a crontabber app. 

[1] https://github.com/mozilla/socorro/blob/master/socorro/cron/base.py#L114

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/b0a622c0293440ec9ff088c5a66afc361e851b69
fixes bug 848680 - add pig support to crontabber

https://github.com/mozilla/socorro/commit/2d1f2ea6291892cfa6e88776992f999eb52aeca3
Merge pull request #1121 from peterbe/bug848680-add-pig-support-to-crontabber

bug 848680 - add pig support to crontabber