Closed Bug 848680 Opened 12 years ago Closed 11 years ago

add pig support to crontabber


(Socorro :: Backend, task)

Not set


(Not tracked)



(Reporter: rhelmer, Unassigned)


(Whiteboard: [qa-])

crontabber has classes to help with calling postgres stored procedures (PostgresCronApp and friends), we should have something similar for Pig jobs.

This would make bug 847735 more palatable, and easier for us to add new jobs.

There's only one pig job right now ( which needs to call out to the external "pig" executable, and do a couple followup tasks like copy out the report, cleanup temporary files etc.

Making it easy to load the results directly into postgres would be nice to have too, since that's something we have been talked about doing.

Here is an example of what we're doing now:
I think the right thing to do would be to base it on BaseBackfillCronApp [1] and simply execute the job with subprocess. 

I'm pretty poor when it comes to pig, hadoop and stuff with HBase but I can write the job as a crontabber app. 

Commits pushed to master at
fixes bug 848680 - add pig support to crontabber
Merge pull request #1121 from peterbe/bug848680-add-pig-support-to-crontabber

bug 848680 - add pig support to crontabber
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → 47
Whiteboard: [qa-]
You need to log in before you can comment on or make changes to this bug.