570158 - Switch to Gearman (or all-Buildbot) for queued tasks from the web app?

Status: RESOLVED FIXED

(Websites Graveyard :: byob.mozilla.com, enhancement)

Description

[Les Orchard [:lorchard]]

Currently, the web app uses a task queue implemented in PHP and run via launchd/cron. It's kind of over-engineered and not necessarily the best system.

Buildbot should eventually take over most or all of the work needed to build repacks, but there may still be a need for a general work queue from the PHP app.

A better, simpler option for a job queue might be Gearman:

http://gearman.org/

Comment 1

[Les Orchard [:lorchard]]

(Unless, of course, Buildbot can just handle all of it... I know Gearman but not Buildbot, so that's the hammer I've been hitting nails with)

Comment 2

[Les Orchard [:lorchard]]

Bumping to a v1.5 milestone (Q3)

Comment 3

[Les Orchard [:lorchard]]

r74310 has changes to switch to gearman.

Trying out gearman on byob staging, hopefully more reliable than the original message queue.

Comment 4

[Les Orchard [:lorchard]]

This should be working on staging now, and it seems like I fixed the MySQL issues. If/when we go live with it on a prod OS X box, these launchd items become important:

/Library/LaunchDaemons/org.mozilla.gearmand.plist
/Library/LaunchDaemons/org.mozilla.byob_gearman_worker.plist

The first ensures gearman runs as a daemon and starts on reboot, and the second does the same for the BYOB repack worker.

The nice thing about this change is that the web app itself can be moved to a linux cluster sooner, as long as there's an OS X box somewhere that can run the worker. 

And, we can even run multiple workers on multiple OS X boxes, since gearman will distribute tasks to any connected workers that support the appropriate jobs. Buildbot should obviate that, but it'll be nice to have in our back pocket

Comment 5

[Les Orchard [:lorchard]]

Oh, and in terms of QA: Nothing should change, really. If the build process continues working on staging, then this was a success.