1343898 - Update wsgi and crontabs for l10n.stage.webapp.scl3.mozilla.com and l10n.webapp.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Axel Hecht [:Pike]]

I'd like to get a few things changed on l10n.mozilla.org and allizom.org configs:

First, the wsgi configs should pass in python-home like so:

    WSGIDaemonProcess ~{APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=~{APP_NAME} python-path=~{APP_ROOT}:~{APP_ROOT}/venv/lib/python2.7/site-packages python-home=~{APP_ROOT}/venv

Also, change the line of 
    Alias /logs.txt /mnt/logs/logs.txt
to
    Alias /logs /mnt/l10n_shared/logs


That's in /etc/httpd/mozilla/domains/l10n.allizom.org.conf for allizom, and 
/etc/httpd/mozilla/domains/l10n.mozilla.org.conf for prod. Just adding the python-home... at the end.

And then I'd like to get a series of cron jobs run, akin to the following contrab:

5       *       *       *       *       cd /data/www/l10n.mozilla.org/elmo; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py progress
13	*/12	*	*	*	cd /data/www/l10n.mozilla.org/elmo; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py build-retention; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py clean_builds


VMs: l10n.stage.webapp.scl3.mozilla.com for stage/allizom, l10n.webapp.scl3.mozilla.com for prod/mozilla.org

Comment 1

[Axel Hecht [:Pike]]

Anyone that can land these requested changes? It's been a month now.

Comment 2

[Daniel Hartnell [:danielh]]

Hey :pike,

I've just taken ownership and I'm working on completing this work as soon as I can. I'll follow up on this bug today with an update for you.

Comment 3

[Daniel Hartnell [:danielh]]

I have updated the WSGI configs in stage and prod to pass in python-home. I'll be keeping an eye on this for the remainder of the day. In the stage Apache configuration, it was sufficient to add what was requested:

>WSGIDaemonProcess ~{APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=~{APP_NAME} python-path=~{APP_ROOT}:~{APP_ROOT}/venv/lib/python2.7/site-packages python-home=~{APP_ROOT}/venv

In the prod Apache configuration, the following line existed:

>WSGIDaemonProcess ${APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=${APP_NAME} python-path=${APP_ROOT}:${APP_ROOT}/venv/lib/python2.7/site-packages

I was not sure if the syntax change from ${APP_NAME} to ~{APP_NAME} was important so I decided to proceed cautiously and retain the existing configuration. So this is what we have in prod right now:

>WSGIDaemonProcess ${APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=${APP_NAME} python-path=${APP_ROOT}:${APP_ROOT}/venv/lib/python2.7/site-packages python-home=${APP_ROOT}/venv

I also updated the Alias /log* line as requested. I'll be adding the cron jobs next.

Comment 4

[Daniel Hartnell [:danielh]]

The cron jobs have been added to stage and prod as well. The job to run "./manage.py progress" already existed in both environments so I only added the second job.

I've checked the stage and prod sites periodically over the last hour and I have not seen any unexpected behavior so I think we can tentatively consider this a success. I'd like to keep the bug open until tomorrow so I can check the cron jobs and make sure they're executing successfully.

Thanks for your patience while we worked through this.

Comment 5

[Daniel Hartnell [:danielh]]

I wanted to add a note regarding my confusion about ${APP_NAME} and ~{APP_NAME}. It appears that we have remapped mod_define::dollar to "~" on the stage server. My implementation of these changes should be consistent with that and I don't expect to see any issues.

Comment 6

[Axel Hecht [:Pike]]

Thanks, cross-checked the apache configs and removed my hacks that I wanted to get rid off, that works.

Re the cron jobs, took me a while to realize that crontabs -l doesn't list them :-/, found them in /etc/cron.d/elmo now :-)

Can we remove the 

# Bug 1343898
13 */12 * * * root cd /data/www/l10n.allizom.org/elmo; /data/www/l10n.allizom.org/venv/bin/python manage.py build-retention; /data/www/l10n.allizom.org/venv/bin/python manage.py clean_builds

from stage again? Thinking about it, that's more harmful that useful, given that stage doesn't have its own automation.

Comment 7

[Daniel Hartnell [:danielh]]

Thanks for taking a look! I'm glad you found the cron jobs - I certainly could have been more explicit there. I removed that specific job from stage and you should see that change once Puppet picks it up and it's applied on the host. I can validate that the change took effect on the host a little later today.

Comment 8

[Daniel Hartnell [:danielh]]

I've confirmed that the job has been removed. Is there anything else I can help out with, Axel?

Comment 9

[Daniel Hartnell [:danielh]]

Hey Axel,

I'm going to plan on marking this resolved today I think. I think we're in a good state after making these changes. If you do need anything at all, go ahead and re-open this bug and let me know what you need. I'll be happy to jump back into it with you.

Thanks,
Daniel

Comment 10

[Axel Hecht [:Pike]]

Thanks for your help, yes, this is fixed now. Marking so.