551513 - Gearman for SUMO/SUMO staging

Status: RESOLVED INCOMPLETE

(mozilla.org Graveyard :: Server Operations, task)

Description

[James Socol [:jsocol, :james]]

To solve bug 532498 comment 45, we're going to move sending notification emails off the Apache process and onto Gearman workers. We'll need a job server and a way to run workers set up. The versions I've been developing with are:

* Gearman 0.12 (http://gearman.org/index.php?id=download)
* PECL Gearman 0.7.0 (http://pecl.php.net/package/gearman/0.7.0)

Comment 1

[James Socol [:jsocol, :james]]

Shyam, the worker I have now runs from the command line like this:

  php notification.php -d

Where the -d detaches it from the terminal and runs it as a daemon. Are there other things I need to do, like writing out the PID somewhere, or is there a whole other approach I should take?

Comment 2

[James Socol [:jsocol, :james]]

This is blocking the last bug in our 1.5.3 release, which we're planning to push to production on the 30th.

Comment 3

[Shyam Mani [:fox2mike]]

James, 

FWIW, we have gearman-0.11 and I think AMO uses php-pear-net_gearman-0.2.3-1, but we can work around those.

I have the gearmand up and running on support-stage, ping me on IRC to figure out the rest? (f.ex I don't know where this notification.php resides etc..)

Comment 4

[James Socol [:jsocol, :james]]

PEAR Gearman and PECL Gearman are completely different APIs. With gearman-0.11, I think PECL version 0.6.0 should build. I'll find you on IRC and we can talk about getting it set up.

The worker is in scripts/gearman/notification.php. Unfortunately, I wasn't able to make it work without the full webroot/ directory as well.

Comment 5

[Shyam Mani [:fox2mike]]

Alright, Dave made us a php-pecl-gearman-0.7x rpm and it seems to be up and running on stage :

[root@mrapp-stage02 gearman]# php notification.php -d
Starting daemon...
[root@mrapp-stage02 gearman]# ps aux | grep notif
apache    7475  0.0  0.0   8524  2948 ?        S    Mar17   0:00 /usr/sbin/sendmail -t -i [e-mail]
apache   14353  0.0  0.0   8528  2968 ?        S    Feb17   0:00 /usr/sbin/sendmail -t -i [e-mail]
root     20123  2.0  0.3  33296 12372 pts/2    S    22:27   0:00 php notification.php -d

Seems alright. 

I'm planning to put up a SUMO Gearman VM for running gearman in prod, probably pm-gearman-sumo01 which I'll setup before the 30th. 

Does this need a full sumo checkout to work? 

Also, leaving this open for prod work.

Comment 6

[James Socol [:jsocol, :james]]

(In reply to comment #5)
> [root@mrapp-stage02 gearman]# php notification.php -d
> Starting daemon...
> [root@mrapp-stage02 gearman]# ps aux | grep notif
> apache    7475  0.0  0.0   8524  2948 ?        S    Mar17   0:00
> /usr/sbin/sendmail -t -i [e-mail]
> apache   14353  0.0  0.0   8528  2968 ?        S    Feb17   0:00
> /usr/sbin/sendmail -t -i [e-mail]
> root     20123  2.0  0.3  33296 12372 pts/2    S    22:27   0:00 php
> notification.php -d

The script does output its PID in the file etc/notification.pid (relative to notification.php) if that's helpful for monitoring/keeping it running.

> I'm planning to put up a SUMO Gearman VM for running gearman in prod, probably
> pm-gearman-sumo01 which I'll setup before the 30th. 

Sounds great! Thanks Shyam.

> Does this need a full sumo checkout to work? 

Yeah, unfortunately. I spent a week trying to break that dependency and got almost nowhere.

Comment 7

[Shyam Mani [:fox2mike]]

(In reply to comment #6)
> The script does output its PID in the file etc/notification.pid (relative to
> notification.php) if that's helpful for monitoring/keeping it running.

Can we change that to /tmp or something? Don't really need a etc in the middle of nowhere and the process should be able to write to tmp anyway.

Also, that means we can monitor /tmp/notification.pid and the process to make sure this doesn't die once it's running (on production) or atleast we get paged when it dies.

So does this work fine on staging?

Comment 8

[James Socol [:jsocol, :james]]

(In reply to comment #7)
> Can we change that to /tmp or something? Don't really need a etc in the middle
> of nowhere and the process should be able to write to tmp anyway.

As of r64861, it now writes the PID to /tmp/notification.pid

> So does this work fine on staging?

I'm not seeing edit notifications. I forgot to mention the new $gearman_servers array in webroot/db/local.php, so I'm hoping you miraculously noticed that change anyway? (See webroot/db/local.php.dist for the format. Both local.php on the web server and on the worker will need to have that set.)

I am seeing email notifications that _don't_ go through Gearman, like the "edit approved" notifications.

Comment 9

[Shyam Mani [:fox2mike]]

James,

I'm apparently not that awesome yet, as I failed to notice the change in local.php :p must......try....harder......

Anyway, configs updated and the daemon has been kicked, let me know if you see notifications fine.

I've verified that the pid file now resides in tmp.

Comment 10

[James Socol [:jsocol, :james]]

(In reply to comment #9)
> I'm apparently not that awesome yet, as I failed to notice the change in
> local.php :p must......try....harder......

My fault, I forgot to mention it.

> Anyway, configs updated and the daemon has been kicked, let me know if you see
> notifications fine.

I'm still not seeing email go out. I'll be around late today (Monday, Pacific) so maybe we can try to figure it out over IRC?

Comment 11

[Shyam Mani [:fox2mike]]

So we had a little IRC troubleshooting session yesterday night (PDT) and it seems like there are some issues with the script + DB connections and the way it's hooked up to tiki, which caused the notification daemon to crash.

James is working on this.

In the meanwhile, I have gearman setup for SUMO prod on pm-gearman-sumo01.

Comment 12

[Shyam Mani [:fox2mike]]

Still waiting on James to finish debugging issues.

Comment 13

[Shyam Mani [:fox2mike]]

Reopen when you're ready.