974155 - Migrate affiliates and reps [dev|stage] to generic cluster

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Brandon Burton [:solarce]]

This is a bug for moving affiliates [dev|stage] from the engagement cluster to the generic cluster. This work is being done as the engagement cluster is no longer highly utilized and the hardware is end-of-life.

We'll start with -dev and work with dev to test via /etc/hosts before cutting over DNS

* https://mana.mozilla.org/wiki/display/websites/Site+Cluster+Migration+Checklist provides an overview of the work to be performed

Once this work is complete we'll migrate prod in bug 964325

Comment 1

[Brandon Burton [:solarce]]

Adding reps to this bug since a lot of the puppet work is the same for both, might as well make the changes for both at once

Comment 2

[Brandon Burton [:solarce]]

-> % svn diff
Index: genericrhel6.pp
===================================================================
--- genericrhel6.pp	(revision 82653)
+++ genericrhel6.pp	(working copy)
@@ -183,6 +183,7 @@

     webapp::logdir {
     [
+        'affiliates.mozilla.org',
         'allizom.org',
         'air.mozilla.org',
         'badges.mozilla.org',
@@ -229,6 +230,7 @@
         'pto.mozilla.org',
         'quality.mozilla.org',
         'releases-api.mozilla.org',
+        'reps.mozilla.org',
         'securitywiki.mozilla.org',
         'summer.mozillapopcorn.org',
         'ship-it.mozilla.org',
bburton@althalus [03:11:45] [~/code/mozilla/sysadmins/puppet/trunk/modules/webapp/manifests]
-> % svn ci -m "adding weblog directories for affiliates and reps, bug 974155"
Sending        manifests/genericrhel6.pp
Transmitting file data .
Committed revision 82655.

-> % svn ci -m "adding affiliates and reps pushbot settings, bug 974155"
Sending        pushbots/genericrhel6-pushbot-settings.js.erb
Transmitting file data .
Committed revision 82657.

-> % svn add genericrhel6-dev/etc-httpd/domains/affiliates-dev.allizom.org.conf genericrhel6-dev/etc-httpd/domains/reps-dev.allizom.org.conf
A         genericrhel6-dev/etc-httpd/domains/affiliates-dev.allizom.org.conf
A         genericrhel6-dev/etc-httpd/domains/reps-dev.allizom.org.conf
bburton@althalus [03:25:15] [~/code/mozilla/sysadmins/puppet/trunk/modules/webapp/files]
-> % svn ci -m "adding reps and affiliates -dev httpd configs, bug 974155"
Adding         files/genericrhel6-dev/etc-httpd/domains/affiliates-dev.allizom.org.conf
Adding         files/genericrhel6-dev/etc-httpd/domains/reps-dev.allizom.org.conf
Transmitting file data ..
Committed revision 82658.

Comment 3

[Brandon Burton [:solarce]]

[[e-mail]1 ~]# rsync -avP engagementadm.private.phx1.mozilla.com:/data/engagement-dev/src/ /data/genericrhel6-dev/src/

bburton@althalus [04:39:12] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "adding affiliates and reps dev/stage celery bits to genericrhel6 cluster, bug 974155"
Sending        trunk/manifests/nodes/celery.pp
Adding         trunk/modules/celery/manifests/genericrhel6/affiliates
Adding         trunk/modules/celery/manifests/genericrhel6/affiliates/dev.pp
Adding         trunk/modules/celery/manifests/genericrhel6/affiliates/stage.pp
Adding         trunk/modules/celery/manifests/genericrhel6/reps
Adding         trunk/modules/celery/manifests/genericrhel6/reps/dev.pp
Adding         trunk/modules/celery/manifests/genericrhel6/reps/stage.pp
Transmitting file data .....
Committed revision 82667.

-> % svn ci -m "adding affiliates/reps dev/stage rabbitmq permissions, bug 974155"
Sending        trunk/manifests/nodes/rabbit.pp
Transmitting file data .
Committed revision 82670.

* Migrated the Chief configs and restarted Chief

Comment 4

[Brandon Burton [:solarce]]

* Application configs updates
* commander_settings.py updates

Remaining tasks:

* cron jobs in puppet
* test deploy with chief
* hand off to dev for testing

Comment 5

[Brandon Burton [:solarce]]

(In reply to Brandon Burton [:solarce] from comment #4)
> * Application configs updates
> * commander_settings.py updates
> 
> Remaining tasks:
> 
> * cron jobs in puppet

bburton@althalus [07:17:41] [~/code/mozilla/sysadmins/puppet/trunk/modules/webapp/files]
-> % svn ci -m "moving affiliates and reps cron jobs to generic cluster, bug 974155"
Adding         files/genericrhel6/admin/etc-cron.d/affiliates
Adding         files/genericrhel6/admin/etc-cron.d/reps.mozilla.org
Transmitting file data ..
Committed revision 82706.

> * test deploy with chief

success: http://genericadm.private.phx1.mozilla.com/chief/affiliates.dev/logs/master.1392827739
> * hand off to dev for testing

:mkelly, can you do some quick checks with the following entry in your /etc/hosts file?

63.245.217.82 affiliates-dev.allizom.org

Comment 6

[Brandon Burton [:solarce]]

(In reply to Brandon Burton [:solarce] from comment #5)
> (In reply to Brandon Burton [:solarce] from comment #4)

> > * test deploy with chief
> 
> success:
> http://genericadm.private.phx1.mozilla.com/chief/affiliates.dev/logs/master.
> 1392827739

I updated the cron job and shell script to call Chief so now affiliates-dev has Chief, same password as stage

Comment 7

[Brandon Burton [:solarce]]

Beginning work on migrating reps-dev now

Comment 8

[Osmose [:osmose, :mkelly]]

(In reply to Brandon Burton [:solarce] from comment #5)
> :mkelly, can you do some quick checks with the following entry in your
> /etc/hosts file?

Site seems up and working fine. :D

Comment 9

[Brandon Burton [:solarce]]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #8)
> (In reply to Brandon Burton [:solarce] from comment #5)
> > :mkelly, can you do some quick checks with the following entry in your
> > /etc/hosts file?
> 
> Site seems up and working fine. :D

Thanks, DNS has been updated, confirmed working, I'll disable the old site shortly, as I begin work on the staging site, just to flush out any issues

* side-note, I did properly configure the celeryd bits as part of the migration, so poke at those at your leisure and let me know if anything seems amiss

Comment 10

[Osmose [:osmose, :mkelly]]

So I missed the emails at first but we've been getting emails from Affiliates-dev since 7:45 AM Pacific, every 15 minutes (they correspond with Affiliates-dev push times):

/data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/lib/python/nose/plugins/manager.py:405: UserWarning: Module memcache was already imported from /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/lib/python/memcache.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
/data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/lib/python/nose/plugins/manager.py:405: UserWarning: Module statsd was already imported from /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor-local/src/pystatsd/statsd/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources
/data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/lib/python/nose/plugins/manager.py:405: UserWarning: Module south was already imported from /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor-local/lib/python/south/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path
  import pkg_resources

I'm also getting an email like this right before that:

/usr/bin/flock: /data/genericrhel6-dev/src/affiliates-dev.allizom.org/update: Success

Each of the emails seems to correspond with a cron job running (two for the leaderboard updates and one for the update). Are they being sent because of output on stderr or something?

Comment 11

[Brandon Burton [:solarce]]

(In reply to Michael Kelly [:mkelly,:Osmose] from comment #10)
> So I missed the emails at first but we've been getting emails from
> Affiliates-dev since 7:45 AM Pacific, every 15 minutes (they correspond with
> Affiliates-dev push times):
> 
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/
> lib/python/nose/plugins/manager.py:405: UserWarning: Module memcache was
> already imported from
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/
> lib/python/memcache.pyc, but /usr/lib/python2.6/site-packages is being added
> to sys.path
>   import pkg_resources
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/
> lib/python/nose/plugins/manager.py:405: UserWarning: Module statsd was
> already imported from
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor-
> local/src/pystatsd/statsd/__init__.pyc, but /usr/lib/python2.6/site-packages
> is being added to sys.path
>   import pkg_resources
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor/
> lib/python/nose/plugins/manager.py:405: UserWarning: Module south was
> already imported from
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/affiliates-app/vendor-
> local/lib/python/south/__init__.pyc, but /usr/lib/python2.6/site-packages is
> being added to sys.path
>   import pkg_resources
> 
> I'm also getting an email like this right before that:
> 
> /usr/bin/flock:
> /data/genericrhel6-dev/src/affiliates-dev.allizom.org/update: Success
> 
> Each of the emails seems to correspond with a cron job running (two for the
> leaderboard updates and one for the update). Are they being sent because of
> output on stderr or something?

I've updated crontab to eat this spurious stdout output

Comment 12

[Brandon Burton [:solarce]]

reps-dev.allizom.org has been migrated

* puppet updates
 - apache config
 - crontab
 - manifest bits for weblogs
* copied /src/ directory to genericadm
* dumped and moved the database to dev1.db.phx1
* no NFS used on dev/stage, just prod for a log file of a cron job
* rabbitmq config pushed with puppet and django config updated
* chief config copied
 * chief push works: http://genericadm.private.phx1.mozilla.com/chief/reps.dev/logs/master.1392855721
* celeryd copied and deployed with puppet
* local.py updated with db, memcache, celery configs
* commander_settings.py updated and confirmed working with chief push above
* cronjobs are running as expected
* deploy worked as noted above
* dns switched

--------

Remaining sites for this bug:

* affiliates.allizom.org
* reps.allizom.org

Comment 13

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Brandon Burton [:solarce] from comment #12)
> reps-dev.allizom.org has been migrated

Mind fixing the 403?

Forbidden

You don't have permission to access /error/noindex.html on this server.

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Stephen Donner [:stephend] from comment #13)
> (In reply to Brandon Burton [:solarce] from comment #12)
> > reps-dev.allizom.org has been migrated
> 
> Mind fixing the 403?
> 
> Forbidden
> 
> You don't have permission to access /error/noindex.html on this server.

i.e. see also https://bugzilla.mozilla.org/show_bug.cgi?id=974556#c3

Comment 15

[Brandon Burton [:solarce]]

(In reply to Stephen Donner [:stephend] from comment #14)
> (In reply to Stephen Donner [:stephend] from comment #13)
> > (In reply to Brandon Burton [:solarce] from comment #12)
> > > reps-dev.allizom.org has been migrated
> > 
> > Mind fixing the 403?
> > 
> > Forbidden
> > 
> > You don't have permission to access /error/noindex.html on this server.
> 
> i.e. see also https://bugzilla.mozilla.org/show_bug.cgi?id=974556#c3

Argh! same issue! my /etc/hosts entry fooled me, clearly I don't make enough DNS entries as I forget there is no view set by default

DNS should update in 10-20 minutes

Comment 16

[John Giannelos [:nemo-yiannis]]

It seems like the cronjob for the automated pull from github master branch is not working properly in reps-dev.alizom.org. At the moment, reps-dev.m.o is a commit behind master.

Can you verify that the cronjob is running?

Thanks!

Comment 17

[Brandon Burton [:solarce]]

Fixed in bug 975341

Comment 18

[Brandon Burton [:solarce]]

affiliates.allizom.org done:

* puppet updates
 - apache config
 - crontab
 - manifest bits for weblogs
* copied /src/affiliates.allizom.org directory to genericadm
* dumped and moved the database to dev1.db.phx1
* NFS for user uploads migrated and content copied
* rabbitmq config pushed with puppet and django config updated
* chief config copied
 * chief push works: http://genericadm.private.phx1.mozilla.com/chief/affiliates.stage/logs/master.1393437337
* celeryd manifests copied and deployed with puppet
* local.py updated with db, memcache, celery configs
* commander_settings.py updated and confirmed working with chief push above
* cronjobs are running as expected
* deploy worked as noted above
* dns switched

Remaining items:
----------------

* reps.allizom.org

Comment 19

[Brandon Burton [:solarce]]

reps.allizom.org done:

* puppet updates
 - apache config
 - crontab
 - manifest bits for weblogs
* copied /src/affiliates.allizom.org directory to genericadm
* dumped and moved the database to dev1.db.phx1
* NFS not used on stage
* rabbitmq config pushed with puppet and django config updated
* chief config copied
 * chief push works: http://genericadm.private.phx1.mozilla.com/chief/reps.stage/logs/master.1393452323
* celeryd manifests copied and deployed with puppet
* local.py updated with db, memcache, celery configs
* commander_settings.py updated and confirmed working with chief push above
* cronjobs are running as expected
* deploy worked as noted above
* dns switched

Now for some remaining clean up

Comment 20

[Brandon Burton [:solarce]]

Cleanup will be tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=964338