1024619 - put both virtualenvs on django apache config

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Robert Helmer [:rhelmer]]

Bug 965590 unifies our virtualenvs to one. Right now this is working OK on stage because we still have the old virtualenv hanging around in our Jenkins workspace, but as soon as we do a clean build this will go away.

The simple way to do this is replace:
WSGIPythonHome /data/socorro/webapp-django/virtualenv/

With:
WSGIPythonPath /data/socorro/application:/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/:/data/socorro/webapp-django/virtualenv/lib/python2.6/site-packages

This only needs to be done for the Django apache config.

After this change ships and sticks we can simplify it to:

WSGIPythonHome /data/socorro/socorro-virtualenv/

Comment 1

[Robert Helmer [:rhelmer]]

Making sure it works OK on stage first:

Sending        files/stage/web/etc-httpd/domains/crash-stats.allizom.org.conf
Transmitting file data .
Committed revision 89003.

Comment 2

[Robert Helmer [:rhelmer]]

Ah apparently WSGIPythonPath isn't honored when in daemon mode:

https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/modwsgi/#using-mod-wsgi-daemon-mode

So, passing python-path to WSGIDaemonProcess and using the global WSGIPythonHome is the way to go, just tested and this works on stage:

Sending        files/stage/web/etc-httpd/domains/crash-stats.allizom.org.conf
Transmitting file data .
Committed revision 89008.

Comment 3

[Robert Helmer [:rhelmer]]

I am out this afternoon, I will push this to prod once I am 100% sure it's working on stage - it can go out anytime ahead of the release since it puts both the old and new virtualenvs on the PYTHONPATH.

Comment 4

[Robert Helmer [:rhelmer]]

Let's also wipe the workspace and make sure things continue to work, so it's not a surprise later if it doesn't...

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> Let's also wipe the workspace and make sure things continue to work, so it's
> not a surprise later if it doesn't...

Done on https://ci.mozilla.org/job/socorro-release/

Comment 6

[Robert Helmer [:rhelmer]]

Modified production Apache configs, all they needed was this at the top:

WSGIPythonHome /data/www/crash-stats.mozilla.org/socorro/socorro-virtualenv/

So the current configs work with the new release and should work with the next one too.

Comment 7

[Robert Helmer [:rhelmer]]

Ah found a place where we're using the django virtualenv, during deploy - this needs to be fixed:

http://socorroadm.private.phx1.mozilla.com/chief/socorro.stage/logs/1115.1402687501

[2014-06-13 19:25:01] Running deploy
[2014-06-13 19:25:01] Running update_stage
[2014-06-13 19:25:01] [socorroadm.stage.private.phx1.mozilla.com] running: /root/bin/auto-update.sh
[2014-06-13 19:25:16] [socorroadm.stage.private.phx1.mozilla.com] failed: /root/bin/auto-update.sh (15.581s)
[socorroadm.stage.private.phx1.mozilla.com] out: Installing 1115 on staging
[socorroadm.stage.private.phx1.mozilla.com] out: ERROR: unable to run database migrations
[socorroadm.stage.private.phx1.mozilla.com] err: /root/bin/auto-update.sh: line 102: /data/socorro/webapp-django/virtualenv/bin/python: No such file or directory
[2014-06-13 19:25:16] Finished update_stage (15.582s)
[2014-06-13 19:25:16] Finished deploy (15.583s)

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #7)
> Ah found a place where we're using the django virtualenv, during deploy -
> this needs to be fixed:
> 
> http://socorroadm.private.phx1.mozilla.com/chief/socorro.stage/logs/1115.
> 1402687501

Fixed in:
Sending        files/stage/admin/scripts/auto-update.sh
Transmitting file data .
Committed revision 89075.

I'll run puppet and push the build to make sure it works.

Lonnen - do you know if prod runs syncdb anywhere? I can't find it, the deploy process looks pretty different there.

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #8)
> (In reply to Robert Helmer [:rhelmer] from comment #7)
> > Ah found a place where we're using the django virtualenv, during deploy -
> > this needs to be fixed:
> > 
> > http://socorroadm.private.phx1.mozilla.com/chief/socorro.stage/logs/1115.
> > 1402687501
> 
> Fixed in:
> Sending        files/stage/admin/scripts/auto-update.sh
> Transmitting file data .
> Committed revision 89075.
> 
> I'll run puppet and push the build to make sure it works.
> 
> Lonnen - do you know if prod runs syncdb anywhere? I can't find it, the
> deploy process looks pretty different there.

Per IRC, this is manual right now.

Comment 10

[Robert Helmer [:rhelmer]]

OK looks like stage deploy is working:
http://socorroadm.private.phx1.mozilla.com/chief/socorro.stage/logs/6589c0abed1d5dc0ec232d2f728c71e5582b50a8.1402691530

crash-stats.allizom.org is on the latest release, and still happy. I verified that the Jenkins workspace no longer contains ./webapp-django/virtualenv

Comment 11

[Robert Helmer [:rhelmer]]

This change was not applied correctly to production, Selena fixed it:

    selena@zhongzong:moz-puppet #509 14:11 :) svn diff modules/webapp/files/crashstats/etc-httpd/domains/crash-stats.mozilla.org.conf
    Index: modules/webapp/files/crashstats/etc-httpd/domains/crash-stats.mozilla.org.conf
    ===================================================================
    --- modules/webapp/files/crashstats/etc-httpd/domains/crash-stats.mozilla.org.conf      (revision 89106)
    +++ modules/webapp/files/crashstats/etc-httpd/domains/crash-stats.mozilla.org.conf      (working copy)
    @@ -1,4 +1,5 @@
     # vim: syntax=apache
    +WSGIPythonHome /data/www/crash-stats.mozilla.org/socorro/socorro-virtualenv
     
     <VirtualHost *:80 *:81>
     # Temp change till SSL cert is in place
    @@ -16,7 +17,7 @@
         Alias /static/ "/data/www/crash-stats.mozilla.org/socorro/webapp-django/static/"
     
         # umask; see bug 1010327
    -    WSGIDaemonProcess crashstats_prod processes=16 threads=4 maximum-requests=1000 display-name=crashstats_prod umask=0002
    +    WSGIDaemonProcess crashstats_prod processes=16 threads=4 maximum-requests=1000 display-name=crashstats_prod umask=0002 python-path=/data/socorro/application:/data/socorro/socorro-virtualenv/lib/python2.6/site-packages/:/data/socorro/webapp-django/virtualenv/lib/python2.6/site-packages
         WSGIProcessGroup crashstats_prod
         WSGIScriptAlias / /data/www/crash-stats.mozilla.org/socorro/webapp-django/wsgi/socorro-crashstats.wsgi process-group=crashstats_prod application-group=crashstats_prod
     
    selena@zhongzong:moz-puppet #511 14:12 :( svn commit -m "Fix for socorro push 20140623 - WSGIPythonHome and WSGIDaemonProcess changed" modules/webapp/files/crashstats/etc-httpd/domains/crash-stats.mozilla.org.conf
    Sending        modules/webapp/files/crashstats/etc-httpd/domains/crash-stats.mozilla.org.conf
    Transmitting file data .
    Committed revision 89476.