1006784 - [Air Mozilla] mod_wsgi sometimes not restarted

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Peter Bengtsson [:peterbe]]

It's happened a couple of times that I kick of Chief to restart dev, stage or prod but it's not actually restarting mod_wsgi.

The file https://air.mozilla.org/media/revision gets updated but when I actually go an look at the site it hasn't restarted. 

As a stop-gap solution can we update the deployment script [0] so it doesn't do the `git rev-parse` [1] part until we know for certain that mod_wsgi was restarted.

I know this is hard to debug because every time this has happened, I get nothing in the Chief logs. 
Yesterday for example, it definitely did the git pull because the template files had been updated but the python code wasn't restarted so the new templates caused a 500 error because it didn't match.

[0] https://github.com/mozilla/airmozilla/blob/master/bin/update/update.py
[1] https://github.com/mozilla/airmozilla/blob/master/bin/update/update.py#L83

Comment 1

[Chris Turra [:cturra]]

updating the update.py script should give you the ability to do the httpd restarts you need. 

you can always see details logs of your deploys in the logs/ directory of chief. the following is an example from air.dev:

  http://genericadm.private.phx1.mozilla.com/chief/air.dev/logs/?C=M;O=D

Comment 2

[Peter Bengtsson [:peterbe]]

But what do you recommend? It currently does `ctx.remote('/bin/touch %s' % settings.REMOTE_WSGI)` [0]

I'm familiar three ways to do this:

1. /etc/init.d/apache2 restart
2. apache2ctl graceful
3. touch /where/is/wsgi/file

It seems that option 3 is the general preference here at Mozilla. I wonder if it's enough. Clearly it sometimes fails. What about the graceful script? Do you think that's a good idea?

[0] https://github.com/mozilla/airmozilla/blob/master/bin/update/update.py#L63

Comment 3

[Chris Turra [:cturra]]

generally, touching the wsgi file is fine, but other projects that have had similar issues you're reporting have gone with the graceful restart approach.

Comment 4

[Peter Bengtsson [:peterbe]]

Cool. So how do I tranform https://github.com/mozilla/airmozilla/blob/master/bin/update/update.py#L63 to do the apache2ctl graceful? Just `ctx.remote('apache2ctl graceful')`?

Comment 5

[Chris Turra [:cturra]]

(In reply to Peter Bengtsson [:peterbe] from comment #4)
> Cool. So how do I tranform
> https://github.com/mozilla/airmozilla/blob/master/bin/update/update.py#L63
> to do the apache2ctl graceful? Just `ctx.remote('apache2ctl graceful')`?

seems reasonable, tho, i generally prefer to use an absolute patch when referencing objects like that. i would do something like the following instead:

  ctx.remote('/etc/init.d/httpd graceful')

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/airmozilla

https://github.com/mozilla/airmozilla/commit/d8eb5dca57536498c198c6d273a34dad7d0b5043
bug 1006784 - [Air Mozilla] mod_wsgi sometimes not restarted

Comment 7

[Peter Bengtsson [:peterbe]]

I've made the change to the deploy script but I can't tell if the server got restarted. 
I could make a silly code change and commit that that shows something. 
Is there a way to tell if the air-dev server actually got restarted?

Comment 8

[Peter Bengtsson [:peterbe]]

Sure enough. It works. 
I had a silly page https://air-dev.allizom.org/__debugger__/ and it does come alive from a Chief push. So clearly the graceful restart worked. 

Let's hope it works better.