600444 - mod_wsgi apps are down on mrapp-stage02

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

http://input.stage.mozilla.com/ is face-palming, right now, along with http://m.input.stage.mozilla.com/, and by face-palming, I mean it's just a white screen returning 200 OK to my GET requests.

Comment 1

[Shyam Mani [:fox2mike]]

Not a blocker, after business hours in MV.

Comment 2

[Shyam Mani [:fox2mike]]

Jeff poked around and found the probable reason why, he'll update the bug and we'll toss it over to the input devs.

Comment 3

[Shyam Mani [:fox2mike]]

So this is a slightly bigger issue. Aravind installed mod_python on mrapp-stage02 earlier today :

Sep 28 16:07:17 Installed: mod_python-3.2.8-3.1.i386

Which hosed mod_wsgi, apparently they can't co-exist :

http://www.g-loaded.eu/2010/04/07/mod_wsgi-incompatible-with-mod_python/

We (IT) will re-convene in the morning and figure out next steps and keep you guys posted. If I remove mod_python, the wsgi apps work fine.

Comment 4

[Shyam Mani [:fox2mike]]

(In reply to comment #3)
> So this is a slightly bigger issue. Aravind installed mod_python on
> mrapp-stage02 earlier today :

Forgot to mention this was for crash-reports.

Comment 5

[Shyam Mani [:fox2mike]]

Also, comment #2 is invalid, since this hasn't really got anything to do with the app itself. Jeff will probably talk to Dave about the issues he found with the app, though.

Comment 6

[Shyam Mani [:fox2mike]]

So I tried upgrading to mod_python 3.3.1 and it still doesn't work. Seems like splitting stuff up is the way to go.

http://code.google.com/p/modwsgi/wiki/InstallationIssues#Incompatible_ModPython_Versions

I'm curious as to why crash-reports wants to use mod_python vs mod_wsgi?

Comment 7

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

Socorro (Crash Reporter) is in a stake of flux.  It has been rolled back from v 1.8 to v 1.7 in both stage and production.  1.7 requires mod_python; 1.8 requires mod_wsgi.  Once we rollout an intermediary version (1.7.4) to production, we'll be able to resume the 1.8 migration and move from mod_python to mod_wsgi on stage.

Comment 8

[Laura Thomson :laura]

The ETA for that, by the way, is 2-3 weeks.

Comment 9

[Shyam Mani [:fox2mike]]

(In reply to comment #3)

> We (IT) will re-convene in the morning and figure out next steps and keep you
> guys posted. If I remove mod_python, the wsgi apps work fine.

mod_python is gone from mrapp-stage02, but I'm seeing an ISE from input, probably a separate issue.

Aravind will be moving crash-reports stage elsewhere so mod_python and mod_wsgi don't clash.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #9)
> (In reply to comment #3)
> 
> > We (IT) will re-convene in the morning and figure out next steps and keep you
> > guys posted. If I remove mod_python, the wsgi apps work fine.
> 
> mod_python is gone from mrapp-stage02, but I'm seeing an ISE from input,
> probably a separate issue.

I spun this off as bug 600667.

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED.