Closed Bug 1343898 Opened 4 years ago Closed 4 years ago

Update wsgi and crontabs for l10n.stage.webapp.scl3.mozilla.com and l10n.webapp.scl3.mozilla.com

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Pike, Assigned: danielh)

References

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/4346])

I'd like to get a few things changed on l10n.mozilla.org and allizom.org configs:

First, the wsgi configs should pass in python-home like so:

    WSGIDaemonProcess ~{APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=~{APP_NAME} python-path=~{APP_ROOT}:~{APP_ROOT}/venv/lib/python2.7/site-packages python-home=~{APP_ROOT}/venv

Also, change the line of 
    Alias /logs.txt /mnt/logs/logs.txt
to
    Alias /logs /mnt/l10n_shared/logs


That's in /etc/httpd/mozilla/domains/l10n.allizom.org.conf for allizom, and 
/etc/httpd/mozilla/domains/l10n.mozilla.org.conf for prod. Just adding the python-home... at the end.

And then I'd like to get a series of cron jobs run, akin to the following contrab:

5       *       *       *       *       cd /data/www/l10n.mozilla.org/elmo; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py progress
13	*/12	*	*	*	cd /data/www/l10n.mozilla.org/elmo; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py build-retention; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py clean_builds


VMs: l10n.stage.webapp.scl3.mozilla.com for stage/allizom, l10n.webapp.scl3.mozilla.com for prod/mozilla.org
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/4346]
Depends on: 1323771
Assignee: server-ops-webops → rwatson
Assignee: rwatson → server-ops-webops
Anyone that can land these requested changes? It's been a month now.
Assignee: server-ops-webops → dhartnell
Hey :pike,

I've just taken ownership and I'm working on completing this work as soon as I can. I'll follow up on this bug today with an update for you.
I have updated the WSGI configs in stage and prod to pass in python-home. I'll be keeping an eye on this for the remainder of the day. In the stage Apache configuration, it was sufficient to add what was requested:

>WSGIDaemonProcess ~{APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=~{APP_NAME} python-path=~{APP_ROOT}:~{APP_ROOT}/venv/lib/python2.7/site-packages python-home=~{APP_ROOT}/venv

In the prod Apache configuration, the following line existed:

>WSGIDaemonProcess ${APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=${APP_NAME} python-path=${APP_ROOT}:${APP_ROOT}/venv/lib/python2.7/site-packages

I was not sure if the syntax change from ${APP_NAME} to ~{APP_NAME} was important so I decided to proceed cautiously and retain the existing configuration. So this is what we have in prod right now:

>WSGIDaemonProcess ${APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=${APP_NAME} python-path=${APP_ROOT}:${APP_ROOT}/venv/lib/python2.7/site-packages python-home=${APP_ROOT}/venv

I also updated the Alias /log* line as requested. I'll be adding the cron jobs next.
The cron jobs have been added to stage and prod as well. The job to run "./manage.py progress" already existed in both environments so I only added the second job.

I've checked the stage and prod sites periodically over the last hour and I have not seen any unexpected behavior so I think we can tentatively consider this a success. I'd like to keep the bug open until tomorrow so I can check the cron jobs and make sure they're executing successfully.

Thanks for your patience while we worked through this.
Flags: needinfo?(l10n)
I wanted to add a note regarding my confusion about ${APP_NAME} and ~{APP_NAME}. It appears that we have remapped mod_define::dollar to "~" on the stage server. My implementation of these changes should be consistent with that and I don't expect to see any issues.
Thanks, cross-checked the apache configs and removed my hacks that I wanted to get rid off, that works.

Re the cron jobs, took me a while to realize that crontabs -l doesn't list them :-/, found them in /etc/cron.d/elmo now :-)

Can we remove the 

# Bug 1343898
13 */12 * * * root cd /data/www/l10n.allizom.org/elmo; /data/www/l10n.allizom.org/venv/bin/python manage.py build-retention; /data/www/l10n.allizom.org/venv/bin/python manage.py clean_builds

from stage again? Thinking about it, that's more harmful that useful, given that stage doesn't have its own automation.
Flags: needinfo?(l10n) → needinfo?(dhartnell)
Thanks for taking a look! I'm glad you found the cron jobs - I certainly could have been more explicit there. I removed that specific job from stage and you should see that change once Puppet picks it up and it's applied on the host. I can validate that the change took effect on the host a little later today.
I've confirmed that the job has been removed. Is there anything else I can help out with, Axel?
Flags: needinfo?(dhartnell) → needinfo?(l10n)
Hey Axel,

I'm going to plan on marking this resolved today I think. I think we're in a good state after making these changes. If you do need anything at all, go ahead and re-open this bug and let me know what you need. I'll be happy to jump back into it with you.

Thanks,
Daniel
Thanks for your help, yes, this is fixed now. Marking so.
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(l10n)
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.