940725 - Deploy Pulse bugzilla_simple_shim to bugzilla-dev

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Mark Côté [:mcote]]

I would like to start testing the whole Bugzilla Change Notification System, starting with bugzilla-dev.  Can we deploy the Pulse shim directly on bugzilla-dev?  There isn't much traffic there, so it should be reasonable to put it directly on the machine.  We can tweak the settings to see how frequently we can run it; on my vagrant box, I can easily run it twice a second.  The shim itself loops to avoid the start-up cost of the Python interpreter; run it with the -r option to enable looping.

The shim is called "bugzilla_simple_shim.py" and is with the rest of the Pulse shims in https://hg.mozilla.org/automation/pulseshims/.  There is a requirements file in there with the required Python packages.  There is also an example config.ini file.  Only the "bugzilla simple" section is required.

Let me know if you have questions about the Pulse configuration.

Comment 1

[Mark Côté [:mcote]]

Something you can help out with? :)

Comment 2

[Kendall Libby [:fubar] (he/him)]

So, I'm struggling through trying to figure out what actually needs to be done to make this go, since our pulse documentation and the puppet config is less than clear. 

I assume the shim will need to talk to something other than the bugzilla db; is that documented anywhere? (For that matter, do we have docs on what pulse does and how it does it? I'm not exactly excited to own it, but there's a real dearth of knowledge around it in IT.)

Comment 3

[Mark Côté [:mcote]]

Yeah wasn't super clear who was going to own it in IT either; we briefed jakem, solarce, and cyliang on it the other day, but from a fairly high level.  Anyway for this, we don't need to touch pulse directly.  You'd just need to be able to talk to port 5672 (which is actually open to the net, but I guess maybe you'd use a direct flow).

As for the config, I set up a user on pulse.  Here's the config you'll need for the shim; I'll msg you the password separately.  You'll have to change the engine and host lines; not sure of the internal hostnames or MySQL user/pass.

[bugzilla simple]
engine = mysql://bugs:bugs@bugzilla/bugs
pulse host = pulse.mozilla.org
pulse port = 5672
pulse vhost = /
pulse user = bugzilla_simple
pulse password = ****
pulse timezone = US/Pacific

Sorry for not sending you this originally; I didn't realize how obscure pulse was. :)

Comment 4

[Kendall Libby [:fubar] (he/him)]

Ok, I think I've got the puppet stuff ready to push once the fw hole is opened. Should we create a new mysql user for pulse?

Comment 5

[Mark Côté [:mcote]]

Yes please, with r/w access only to that one table (push_notify).

Comment 6

[Kendall Libby [:fubar] (he/him)]

Attempting to run the shim, I get:

web5.stage.bugs.scl3# /usr/bin/python /data/pulse/pulseshims/bugzilla_simple_shim.py /data/pulse/config.ini 
Traceback (most recent call last):
  File "/data/pulse/pulseshims/bugzilla_simple_shim.py", line 25, in <module>
    from mozillapulse.messages.bugzilla import SimpleBugMessage
ImportError: cannot import name SimpleBugMessage

Python 2.6.6
MozillaPulse .5
pulseshims dc7e9d3a4474

looks like it should be GenericBugMessage, but I'm not certain.

Comment 7

[Mark Côté [:mcote]]

The latest mozillapulse on pypi is 0.80--why is 0.5 being installed?  Admittedly I should have pegged the version in the requirements file... I'll update that.

Comment 8

[Kendall Libby [:fubar] (he/him)]

# pip install --proxy=dc-proxy:3128 mozillapulse -v
Downloading/unpacking mozillapulse
  Using version .5 (newest of versions: .5, .4, 0.80, 0.70, 0.61, 0.6)
  Downloading MozillaPulse-.5.zip

Specifying version works, though:

# pip install --proxy=dc-proxy:3128 mozillapulse==0.80
Downloading/unpacking mozillapulse==0.80
  Downloading MozillaPulse-0.80.tar.gz

Comment 9

[Kendall Libby [:fubar] (he/him)]

Was missing read access to the rest of the bugs db; fixed.  Aaaaand apparently it's working, or at least not throwing errors. :)

The pulse shim stuff in puppet throws it into a cron job run every 5 minutes, which would appear to not be what you'd prefer. How would you like this run?

Comment 10

[Mark Côté [:mcote]]

That's really weird--pip install mozillapulse on my system chooses 0.80.  Anyway I updated the requirements file in the shim to specify >=0.80.

It should just be run as a regular process.  Since I'd like it to run very often--much, much more frequently than every 5 minutes--I made it loop inside the shim to avoid the cost of Python's start-up time.  The -r option controls this; I'd like to see if, say, 5 seconds (-r 5) is reasonable.  Or even lower! (I was running it on my box box with -r 0.5 and didn't see any noticeable problems)

Comment 11

[Kendall Libby [:fubar] (he/him)]

supervisord to the rescue. it's currently running  with '-r 5'.

Comment 12

[Mark Côté [:mcote]]

Cool.  I'm just waiting for the extension to be turned on (bug 949545) to make sure this all works.

Comment 13

[Mark Côté [:mcote]]

Worked fine.  Now to get it into production (bug 952880)! :)