Closed Bug 848680 Opened 9 years ago Closed 9 years ago

add pig support to crontabber

Categories

(Socorro :: Backend, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rhelmer, Unassigned)

Details

(Whiteboard: [qa-])

crontabber has classes to help with calling postgres stored procedures (PostgresCronApp and friends), we should have something similar for Pig jobs.

This would make bug 847735 more palatable, and easier for us to add new jobs.

There's only one pig job right now (https://github.com/mozilla/socorro/blob/master/scripts/crons/cron_modulelist.sh) which needs to call out to the external "pig" executable, and do a couple followup tasks like copy out the report, cleanup temporary files etc.

Making it easy to load the results directly into postgres would be nice to have too, since that's something we have been talked about doing.

Here is an example of what we're doing now:
https://github.com/mozilla/socorro/blob/master/scripts/crons/cron_modulelist.sh
I think the right thing to do would be to base it on BaseBackfillCronApp [1] and simply execute the job with subprocess. 

I'm pretty poor when it comes to pig, hadoop and stuff with HBase but I can write the job as a crontabber app. 

[1] https://github.com/mozilla/socorro/blob/master/socorro/cron/base.py#L114
Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/b0a622c0293440ec9ff088c5a66afc361e851b69
fixes bug 848680 - add pig support to crontabber

https://github.com/mozilla/socorro/commit/2d1f2ea6291892cfa6e88776992f999eb52aeca3
Merge pull request #1121 from peterbe/bug848680-add-pig-support-to-crontabber

bug 848680 - add pig support to crontabber
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → 47
Whiteboard: [qa-]
You need to log in before you can comment on or make changes to this bug.