909900 - Reconfigure socorro[1-2].stage.webapp.phx1 Admin node and Paths

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task, P1)

Description

[Brandon Burton [:solarce]]

Currently socorro[1-2].stage.webapp.phx1 are being managed by socorroadm.private.phx1 and were setup like a typical web cluster for historical reasons around the deployment of the Django crashstats alongside the PHP crashsts.

PHP crashstats has been retired and because of higher level Socorro architectural concerns it's not feasible to share a single admin node at this time.

After discussion with :lonnen we agreed it makes the most sense to migrate socorro[1-2].stage.webapp.phx1 to be managed and deployed to from socorroadm.stage.private.phx1

This involves two major changes
1. Changing the path of the socorro code from /data/www/crash-stats.allizom.org/socorro/ to /data/socorro/ and updating the Apache configs to reflect the path changes
 * I'll do this one web server at a time, disabled in the LB while i work on it, to minimize the disruption to staging
2. Update the deployment scripts to push to the web servers from socorroadm.stage.private.phx1

Comment 1

[Brandon Burton [:solarce]]

I will be disabling the stage auto-deploys while I do this work

Comment 2

[Brandon Burton [:solarce]]

bburton@althalus [01:45:20] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn diff
Index: manifests/nodes/socorro.pp
===================================================================
--- manifests/nodes/socorro.pp	(revision 73928)
+++ manifests/nodes/socorro.pp	(working copy)
@@ -909,14 +909,26 @@
 }

 node /^socorro[12].stage.webapp.phx1.mozilla.com$/ {
+    #class {
+    #    'webapp::base':
+    #        webapp_admin => 'socorroadm.stage.private.phx1.mozilla.com',
+    #        cluster => 'crashstats-stage',
+    #        syslog_host => '10.8.81.64',
+    #        ganglia_addr => '239.2.11.104',
+    #        ganglia_cluster => 'Web';
+    #}
+
     class {
-        'webapp::base':
-            webapp_admin => 'socorroadm.private.phx1.mozilla.com',
-            cluster => 'crashstats-stage',
-            syslog_host => '10.8.81.64',
-            ganglia_addr => '239.2.11.104',
-            ganglia_cluster => 'Web';
-    }
+      'webapp::socorro':
+         cluster         => 'socorro-stage',
+         host_type       => 'web',
+         include_apache  => true,
+         include_python  => false,
+         ganglia_addr    => '239.2.11.104',
+         ganglia_cluster => 'Web',
+         webapp_admin    => 'socorroadm.stage.private.phx1.mozilla.com';
+    }
+
     #include webapp::crashstats::keys
     include webapp::socorro::keys::stage
     include webapp::crashstats::stage
bburton@althalus [01:45:20] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "work to switch socorro stage web to use socorroadm.stage.p.phx1, bug 909900"
Sending        trunk/manifests/nodes/socorro.pp
Transmitting file data .
Committed revision 73984.

Comment 3

[Brandon Burton [:solarce]]

bburton@althalus [07:22:56] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn status
D       modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.com.conf
A       modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.org.conf
bburton@althalus [07:22:58] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "fixing apache config files for socorro stage with merge work, bug 909900"
Deleting       trunk/modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.com.conf
Adding         trunk/modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.org.conf
Transmitting file data .
Committed revision 74040.

Comment 4

[Brandon Burton [:solarce]]

bburton@althalus [08:20:53] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn diff
Index: modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.org.conf
===================================================================
--- modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.org.conf	(revision 74040)
+++ modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.org.conf	(working copy)
@@ -10,11 +10,11 @@

     RewriteEngine On

-    Alias /static/ "/data/www/crash-stats.allizom.org/socorro/webapp-django/static/"
+    Alias /static/ "/data/socorro/webapp-django/static/"

     WSGIDaemonProcess crashstats_stage processes=16 threads=4 maximum-requests=1000 display-name=crashstats_stage
     WSGIProcessGroup crashstats_stage
-    WSGIScriptAlias / /data/www/crash-stats.allizom.org/socorro/webapp-django/wsgi/socorro-crashstats.wsgi process-group=crashstats_stage application-group=crashstats_stage
+    WSGIScriptAlias / /data/socorro/webapp-django/wsgi/socorro-crashstats.wsgi process-group=crashstats_stage application-group=crashstats_stage

     <Location />
         ExpiresActive On
bburton@althalus [08:20:55] [~/code/mozilla/sysadmins/puppet/trunk]
-> % svn ci -m "changing Apache config paths to match rest of socorro stage, bug 909900"
Sending        trunk/modules/webapp/files/socorro-stage/web/etc-httpd/domains/crash-stats.allizom.org.conf
Transmitting file data .
Committed revision 74045.

Comment 5

[Brandon Burton [:solarce]]

Testing of changes is working on socorro1.stage.webapp.phx1, gonna wait until the morning to enable on socorro2 and put into rotation because

1. need bug 910069 for flows so deploys work
2. want to have QA test it out and don't want it to break overnight

Comment 6

[Brandon Burton [:solarce]]

Did last bits of finishing this to get build pushed for QA for prod push :>

Seems to be ok so far, waiting on QA to review it then will do cleanup and RF

Comment 7

[Brandon Burton [:solarce]]

This is done and working fine, we've done multiple stage pushes since these changes without error

Comment 8

[Matt Brandt [:mbrandt]]

QA verified - woot ya to comment 7. We seem stable. Multiple pushes, achievement unlocked