557049 - [amo] monitor zamboni wsgi in prod

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Jeff Balogh (:jbalogh)]

We updated to 5.8.3 on Thursday, but I left out touching the .wsgi file from the upgrade script.  It appears that the code got updated but apache is still running the old code.  We're getting tracebacks that shouldn't be possible anymore, and the line numbers are all wrong.

`touch /data/amo_python/www/services.prod/zamboni/wsgi/zamboni.wsgi` will force mod_wsgi to reload.

`rm **/*.pyc` for extra paranoia.

Comment 1

[Wil Clouser [:clouserw]]

Jeff wrote a short script to check if wsgi is older than the git update:

> from datetime import datetime
> import os
> import subprocess
> import sys
> 
> GIT = "git log -1 --date=local | grep Date: | sed 's/Date: *//'"
> 
> wsgi = datetime.fromtimestamp(os.stat('wsgi/zamboni.wsgi').st_mtime)
> git_ = subprocess.Popen(GIT, stdout=subprocess.PIPE, shell=True).communicate()[0]
> git = datetime.strptime(git_.strip(), '%a %b %d %H:%M:%S %Y')
> 
> if wsgi > git:
>     print "ok"
>     sys.exit(0)
> else:
>     print "wsgi is too old"
>     sys.exit(2)

We could add that to nagios to make sure this doesn't happen again.

Comment 2

[Jeremy Orem [:oremj]]

Apache was restarted after the updated. Even if it isn't updated I have MaxRequestsPerChild set at 100, so the processes should refresh fairly quickly.

Looks like no .pyc files.
/data/bin/issue-multi-command.py -c 1 amo 'find /data/amo_python/www/prod | grep pyc'
===pm-app-amo02===

===pm-app-amo03===

===pm-app-amo04===

===pm-app-amo05===

===pm-app-amo07===

===pm-app-amo08===

===pm-app-amo10===

===pm-app-amo11===

===pm-app-amo12===

===pm-app-amo13===

===pm-app-amo14===

===pm-app-amo15===

===pm-app-amo16===

===pm-app-amo17===

===pm-app-amo18===

===pm-app-amo19===

===pm-app-amo20===

===pm-app-amo21===

===pm-app-amo24===

===pm-app-amo25===

===pm-app-amo26===

Comment 3

[Jeff Balogh (:jbalogh)]

(In reply to comment #2)
> Apache was restarted after the updated. Even if it isn't updated I have
> MaxRequestsPerChild set at 100, so the processes should refresh fairly quickly.

Hmm, it seemed like something was not right.  I haven't seen any timeouts since April 3rd, so we'll see how it goes.

With the MaxRequests, would that re-read the source or make a new fork from the parent mod_wsgi?

> Looks like no .pyc files.

That seems...unlikely.  I don't know why Python wouldn't be creating .pyc files.  

Oh, maybe it doesn't have write perms on those files?

> /data/bin/issue-multi-command.py -c 1 amo 'find /data/amo_python/www/prod |
> grep pyc'

Is the source for that available somewhere?  I'm just curious.

Comment 4

[Jeremy Orem [:oremj]]

(In reply to comment #3)
> (In reply to comment #2)
> > Apache was restarted after the updated. Even if it isn't updated I have
> > MaxRequestsPerChild set at 100, so the processes should refresh fairly quickly.
> 
> Hmm, it seemed like something was not right.  I haven't seen any timeouts since
> April 3rd, so we'll see how it goes.
> 
> With the MaxRequests, would that re-read the source or make a new fork from the
> parent mod_wsgi?
Now that I think about it, this might not reload mod_wsgi at all since it is in daemon mode.

> 
> > Looks like no .pyc files.
> 
> That seems...unlikely.  I don't know why Python wouldn't be creating .pyc
> files.  
> 
> Oh, maybe it doesn't have write perms on those files?

All the directories are owned by root, so it doesn't have write perms.

> 
> > /data/bin/issue-multi-command.py -c 1 amo 'find /data/amo_python/www/prod |
> > grep pyc'
> 
> Is the source for that available somewhere?  I'm just curious.

http://gist.github.com/356796

Comment 5

[Jeremy Orem [:oremj]]

I'm not sure this script will work.  We don't sync out zamboni's .git.  Any other ideas?
(In reply to comment #1)
> Jeff wrote a short script to check if wsgi is older than the git update:
> 
> > from datetime import datetime
> > import os
> > import subprocess
> > import sys
> > 
> > GIT = "git log -1 --date=local | grep Date: | sed 's/Date: *//'"
> > 
> > wsgi = datetime.fromtimestamp(os.stat('wsgi/zamboni.wsgi').st_mtime)
> > git_ = subprocess.Popen(GIT, stdout=subprocess.PIPE, shell=True).communicate()[0]
> > git = datetime.strptime(git_.strip(), '%a %b %d %H:%M:%S %Y')
> > 
> > if wsgi > git:
> >     print "ok"
> >     sys.exit(0)
> > else:
> >     print "wsgi is too old"
> >     sys.exit(2)
> 
> We could add that to nagios to make sure this doesn't happen again.

Comment 6

[Jeff Balogh (:jbalogh)]

We could check that zamboni.wsgi is newer than any other .py or .html files in the tree.

Comment 7

[Jeremy Orem [:oremj]]

For production we typically restart apache instead of touching zamboni.wsgi.  Should we start touching it instead or can you think of another way to do this?

Comment 8

[Wil Clouser [:clouserw]]

We need to restart apache to pick up .mo changes at least until PHP goes away.  I don't know how wsgi+.mo works, does it still need restarting?

Comment 9

[Jeremy Orem [:oremj]]

14:05 < jbalogh> oremj: I just pushed /services/loaded

Comment 10

[Jeff Balogh (:jbalogh)]

This is expected to go to prod on 6/1.

Comment 11

[Jeff Balogh (:jbalogh)]

https://addons.mozilla.org/z/services/loaded

Comment 12

[Jeremy Orem [:oremj]]

Do we want to do something like this?

http://gist.github.com/441263

Comment 13

[Jeremy Orem [:oremj]]

Final monitor: http://gist.github.com/441263


Nagios is monitoring it now.

Comment 14

[Wil Clouser [:clouserw]]

Can you add a comment to the top about what it does and add it to http://github.com/jbalogh/zamboni/tree/master/scripts/ ?  Thanks.

Comment 15

[Jeremy Orem [:oremj]]

http://github.com/jbalogh/zamboni/commit/c19f52ddb474cc2f976f548522e78b6eff678731