775003 - deploy push extension and daemon to production

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[:glob ✱]

we need to deploy the bugzilla push addon and connector for service-now from bug 757702 to production.

in terms of timing, service-now's go-live is on july 23rd, however we'll need more time to prepare this deployment.  i'd love to have something in place for the end of this month if possible.

requirements: the bugzilla push daemon is written as part of the bugzilla framework itself, so the system needs to be deployed the same as a bmo webhead, only without httpd (same version of perl, same perl modules, etc).  memory requirements are significantly lower; 2 to 4 gig should be sufficient.  logs are written to the bmo database, so disk requirements are minimal.

it needs write access to the bmo production database, but no other external connectivity.

the daemon itself doesn't need to run as root, but will need to be in the webserver's group in order to read the bugzilla files.  it would make troubleshooting *significantly* easier if it's an account which we (dkl and i) are able to log in as, so we can restart the daemon, and run it in debug mode.

Comment 1

[:glob ✱]

we could also run the moco-ldap-checking script on this system, as it's a requirement for correct integration with service-now (see bug 757698).

if we choose to do this, this VM will need a flow opened to moco ldap.

Comment 2

[:glob ✱]

(In reply to Byron Jones ‹:glob› from comment #1)
> if we choose to do this, this VM will need a flow opened to moco ldap.

actually, we'll need to be able to connect to moco from the servicenow connector.


i've pushed a skeleton extension to get the schema changes in place without any other code changes - this means we can update the schema without downtime.

Committing to: bzr+ssh://[e-mail]/bmo/4.0/
added extensions/Push
added extensions/Push/Config.pm
added extensions/Push/Extension.pm
Committed revision 8249.

Comment 3

[Shyam Mani [:fox2mike]]

I've gotten the VM going, servicenow1.bugs.phx1.mozilla.com and have a preliminary puppet manifest up too. That needs to be tested and the machine needs to be puppetized.

Comment 4

[:glob ✱]

the project to integration bmo with servicenow has been cancelled, however the extension and daemon was designed to be modular, and will be used for other projects (initially bug 774590).

please continue with this deployment, however it may be a good idea to rename the system from servicenow1 to push1 or similar (the bugzilla extension is called "push").

Comment 5

[:glob ✱]

please deploy the following perl modules to the 'push box':

  Net::SFTP
  XML::Simple

thanks

Comment 6

[:glob ✱]

this box will need access to 180.169.5.248:22

Comment 7

[:glob ✱]

can i please get a timeframe for when this will be ready?

Comment 8

[Shyam Mani [:fox2mike]]

With everything else going on, I'm hoping before the end of the week. I will work on this today and file a flow bug too, once I fix up the hostname etc.

Comment 9

[Shyam Mani [:fox2mike]]

So, now the blocker is ImageMagick-perl. I fixed the issues with perl-Test-Script...oh well. I'll work on this tomm.

Comment 10

[Shyam Mani [:fox2mike]]

All good to go here :)

They moved ImageMagick-perl from base to the optional channel, so I had to add that to the box.

Byron, what are he next steps here?

Comment 11

[:glob ✱]

\o/ thanks!

i've just committed the full push extension to bzr, it'll be pushed to the production webheads tomorrow.

assuming it has access to 180.169.5.248:22, i'll like to ssh into server and run the daemon in debug mode (extensions/Push/bin/bugzilla-pushd.pl -d -f start), and watch it to ensure it's picking up tcl bugs an sending them.

once i'm happy with that, we can configure it to start as a normal daemon, and start it.

Comment 12

[:glob ✱]

you'll also need to update the bmo push process to ensure this system is updated at the same time webheads are updated.

Comment 13

[Shyam Mani [:fox2mike]]

(In reply to Byron Jones ‹:glob› from comment #12)
> you'll also need to update the bmo push process to ensure this system is
> updated at the same time webheads are updated.

Yeah. That'll happen automatically in SCL3. Probably going to be a cronjob here.

Comment 14

[Shyam Mani [:fox2mike]]

:glob ran into some perl module issues here and is working out a plan to fix those up.

Comment 15

[:glob ✱]

(In reply to Shyam Mani [:fox2mike] from comment #14)
> :glob ran into some perl module issues here and is working out a plan to fix
> those up.

i've kludged around the problem... the issue is bmo/4.0/lib contains modules compiled for rhel5, which don't work on rhel6.

i've updated the bugzilla-push daemon to not check that directory for modules, as the new systems have them all system-installed rather than from that directory.

Comment 16

[Shyam Mani [:fox2mike]]

This is now waiting on an NFS volume, I'll work on that today.

Comment 17

[Shyam Mani [:fox2mike]]

(In reply to Shyam Mani [:fox2mike] from comment #16)
> This is now waiting on an NFS volume, I'll work on that today.

Mount is now online. Byron : happy to work on next steps with you.

Comment 18

[:glob ✱]

$ sudo ./extensions/Push/bin/bugzilla-pushd.pl -d -f start
Starting up...
Can't connect to the database.
Error: Unknown MySQL server host 'db-bugs01-rw' (1)

Comment 19

[Shyam Mani [:fox2mike]]

Fixed that...and :

[[e-mail]1 bugzilla.mozilla.org]# ./extensions/Push/bin/bugzilla-pushd.pl -d -f start

Starting up...
[Thu Dec 20 06:25:15 2012] DEBUG: global: set log_purge=7
[Thu Dec 20 06:25:15 2012] DEBUG: global: set enabled=Disabled
[Thu Dec 20 06:25:15 2012] DEBUG: Loading connector 'AMQP'
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set password=********
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set vhost=/
Use of uninitialized value $value in sprintf at /loader/0x2ea06a0/Bugzilla/Extension/Push/Config.pm line 83.
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set queue=
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set port=5672
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set exchange=
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set username=guest
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set host=localhost
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set enabled=Disabled
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set password=********
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set vhost=/
Use of uninitialized value $value in sprintf at /loader/0x2ea06a0/Bugzilla/Extension/Push/Config.pm line 83.
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set queue=
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set port=5672
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set exchange=
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set username=guest
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set host=localhost
[Thu Dec 20 06:25:15 2012] DEBUG: AMQP: set enabled=Disabled
[Thu Dec 20 06:25:15 2012] DEBUG: Loading connector 'File'
[Thu Dec 20 06:25:15 2012] DEBUG: File: set filename=push.log
[Thu Dec 20 06:25:15 2012] DEBUG: File: set enabled=Disabled
[Thu Dec 20 06:25:15 2012] DEBUG: Loading connector 'TCL'
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set sftp_port=22
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set sftp_remote_path=
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set sftp_user=
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set tcl_user=[e-mail]
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set sftp_host=
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set sftp_pass=********
[Thu Dec 20 06:25:15 2012] DEBUG: TCL: set enabled=Disabled
[Thu Dec 20 06:25:15 2012] DEBUG: resetting backoff for AMQP
[Thu Dec 20 06:25:15 2012] DEBUG: resetting backoff for File
[Thu Dec 20 06:25:15 2012] DEBUG: resetting backoff for TCL

I've ctrl+c'd the process. You can run this again :)

Comment 20

[:glob ✱]

thanks shyam, everything looks good \o/


let's leave the system in its current state (ie. deamon not running) until we get the go-head from tcl.  when that happens, i'll configure the extension, run the daemon in debug mode to ensure things are well, then file a new bug to get the daemon configured to run as a init.d service and started.

Comment 21

[Shyam Mani [:fox2mike]]

Sounds like a plan :) Thanks!