Update wsgi and crontabs for l10n.stage.webapp.scl3.mozilla.com and l10n.webapp.scl3.mozilla.com

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
19 days ago

People

(Reporter: Pike, Assigned: danielh)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/4346])

(Reporter)

Description

2 years ago
I'd like to get a few things changed on l10n.mozilla.org and allizom.org configs:

First, the wsgi configs should pass in python-home like so:

    WSGIDaemonProcess ~{APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=~{APP_NAME} python-path=~{APP_ROOT}:~{APP_ROOT}/venv/lib/python2.7/site-packages python-home=~{APP_ROOT}/venv

Also, change the line of 
    Alias /logs.txt /mnt/logs/logs.txt
to
    Alias /logs /mnt/l10n_shared/logs


That's in /etc/httpd/mozilla/domains/l10n.allizom.org.conf for allizom, and 
/etc/httpd/mozilla/domains/l10n.mozilla.org.conf for prod. Just adding the python-home... at the end.

And then I'd like to get a series of cron jobs run, akin to the following contrab:

5       *       *       *       *       cd /data/www/l10n.mozilla.org/elmo; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py progress
13	*/12	*	*	*	cd /data/www/l10n.mozilla.org/elmo; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py build-retention; /data/www/l10n.mozilla.org/venv/bin/python ./manage.py clean_builds


VMs: l10n.stage.webapp.scl3.mozilla.com for stage/allizom, l10n.webapp.scl3.mozilla.com for prod/mozilla.org

Updated

2 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/4346]
(Reporter)

Updated

2 years ago
Depends on: 1323771
(Reporter)

Updated

2 years ago
Blocks: 1304413

Updated

2 years ago
Assignee: server-ops-webops → rwatson
Assignee: rwatson → server-ops-webops
(Reporter)

Comment 1

2 years ago
Anyone that can land these requested changes? It's been a month now.

Updated

2 years ago
Assignee: server-ops-webops → dhartnell
(Assignee)

Comment 2

2 years ago
Hey :pike,

I've just taken ownership and I'm working on completing this work as soon as I can. I'll follow up on this bug today with an update for you.
(Assignee)

Comment 3

2 years ago
I have updated the WSGI configs in stage and prod to pass in python-home. I'll be keeping an eye on this for the remainder of the day. In the stage Apache configuration, it was sufficient to add what was requested:

>WSGIDaemonProcess ~{APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=~{APP_NAME} python-path=~{APP_ROOT}:~{APP_ROOT}/venv/lib/python2.7/site-packages python-home=~{APP_ROOT}/venv

In the prod Apache configuration, the following line existed:

>WSGIDaemonProcess ${APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=${APP_NAME} python-path=${APP_ROOT}:${APP_ROOT}/venv/lib/python2.7/site-packages

I was not sure if the syntax change from ${APP_NAME} to ~{APP_NAME} was important so I decided to proceed cautiously and retain the existing configuration. So this is what we have in prod right now:

>WSGIDaemonProcess ${APP_NAME} processes=2 threads=1 maximum-requests=500 display-name=${APP_NAME} python-path=${APP_ROOT}:${APP_ROOT}/venv/lib/python2.7/site-packages python-home=${APP_ROOT}/venv

I also updated the Alias /log* line as requested. I'll be adding the cron jobs next.
(Assignee)

Comment 4

2 years ago
The cron jobs have been added to stage and prod as well. The job to run "./manage.py progress" already existed in both environments so I only added the second job.

I've checked the stage and prod sites periodically over the last hour and I have not seen any unexpected behavior so I think we can tentatively consider this a success. I'd like to keep the bug open until tomorrow so I can check the cron jobs and make sure they're executing successfully.

Thanks for your patience while we worked through this.
Flags: needinfo?(l10n)
(Assignee)

Comment 5

2 years ago
I wanted to add a note regarding my confusion about ${APP_NAME} and ~{APP_NAME}. It appears that we have remapped mod_define::dollar to "~" on the stage server. My implementation of these changes should be consistent with that and I don't expect to see any issues.
(Reporter)

Comment 6

2 years ago
Thanks, cross-checked the apache configs and removed my hacks that I wanted to get rid off, that works.

Re the cron jobs, took me a while to realize that crontabs -l doesn't list them :-/, found them in /etc/cron.d/elmo now :-)

Can we remove the 

# Bug 1343898
13 */12 * * * root cd /data/www/l10n.allizom.org/elmo; /data/www/l10n.allizom.org/venv/bin/python manage.py build-retention; /data/www/l10n.allizom.org/venv/bin/python manage.py clean_builds

from stage again? Thinking about it, that's more harmful that useful, given that stage doesn't have its own automation.
Flags: needinfo?(l10n) → needinfo?(dhartnell)
(Assignee)

Comment 7

2 years ago
Thanks for taking a look! I'm glad you found the cron jobs - I certainly could have been more explicit there. I removed that specific job from stage and you should see that change once Puppet picks it up and it's applied on the host. I can validate that the change took effect on the host a little later today.
(Assignee)

Comment 8

2 years ago
I've confirmed that the job has been removed. Is there anything else I can help out with, Axel?
Flags: needinfo?(dhartnell) → needinfo?(l10n)
(Assignee)

Comment 9

2 years ago
Hey Axel,

I'm going to plan on marking this resolved today I think. I think we're in a good state after making these changes. If you do need anything at all, go ahead and re-open this bug and let me know what you need. I'll be happy to jump back into it with you.

Thanks,
Daniel
(Reporter)

Comment 10

2 years ago
Thanks for your help, yes, this is fixed now. Marking so.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Flags: needinfo?(l10n)
Resolution: --- → FIXED

Updated

19 days ago
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.