600667 - Input staging is returning a 500 Internal Server Error / down

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Though bug 600444 is fixed, http://input.stage.mozilla.com/ and http://m.input.stage.mozilla.com/ are still down, but now returning a 500 Internal Server Error

Sorry, not sure if this is an app or IT bug.

Comment 1

[Aakash Desai [:aakashd]]

We haven't gotten any word from IT on why staging is returning a ISE. Shyam, in bug 600444, mentioned that this is a separate issue from mod_python and mod_wsgi not playing well together. Since that's been fixed, I'm still unsure why this isn't fixed.

Comment 2

[Shyam Mani [:fox2mike]]

Need a dev to troubleshoot.

[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4] mod_wsgi (pid=11925): Target WSGI script '/data/www/input.stage.mozilla.com/reporter/wsgi/reporter.wsgi' cannot be loaded as Python module.
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4] mod_wsgi (pid=11925): Exception occurred processing WSGI script '/data/www/input.stage.mozilla.com/reporter/wsgi/reporter.wsgi'.
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4] Traceback (most recent call last):
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/wsgi/reporter.wsgi", line 26, in <module>
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     command.validate()
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/core/management/base.py", line 245, in validate
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     num_errors = get_validation_errors(s, app)
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/core/management/validation.py", line 22, in get_validation_errors
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     from django.db import models, connection
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/db/__init__.py", line 75, in <module>
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     connection = connections[DEFAULT_DB_ALIAS]
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/db/utils.py", line 91, in __getitem__
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     backend = load_backend(db['ENGINE'])
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/db/utils.py", line 32, in load_backend
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     return import_module('.base', backend_name)
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/utils/importlib.py", line 35, in import_module
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     __import__(name)
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]   File "/data/www/input.stage.mozilla.com/reporter/vendor/packages/Django/django/db/backends/mysql/base.py", line 14, in <module>
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4]     raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
[Thu Sep 30 07:43:17 2010] [error] [client 10.2.81.4] ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb

MySQLdb is there and installed, is also in the path. Don't know why this is happening.

Comment 3

[Aakash Desai [:aakashd]]

Fred, Dave, Ryan can either you help out Shyam when you've got the cycles?

Comment 4

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

I took a look yesterday with Shyam and couldn't figure out what was going on.  I'll dive in more.

CCing Oremj and Jbalogh in case they've seen this before.

Comment 5

[Shyam Mani [:fox2mike]]

Passing on to Jeremy, I'm gone till Monday for the most part.

Comment 6

[Jeff Balogh (:jbalogh)]

This is sys.path:

'/data/virtualenvs/BASELINE/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg'
'/data/virtualenvs/BASELINE/lib/python2.6/site-packages/pip-0.6.3-py2.6.egg'
'/data/virtualenvs/BASELINE/lib/python26.zip'
'/data/virtualenvs/BASELINE/lib/python2.6'
'/data/virtualenvs/BASELINE/lib/python2.6/plat-linux2'
'/data/virtualenvs/BASELINE/lib/python2.6/lib-tk'
'/data/virtualenvs/BASELINE/lib/python2.6/lib-old'
'/data/virtualenvs/BASELINE/lib/python2.6/lib-dynload'
'/usr/lib/python2.6'
'/usr/lib/python2.6/plat-linux2'
'/data/virtualenvs/BASELINE/lib/python2.6/site-packages'

I think that means WSGIPythonHome is set to /data/virtualenvs/BASELINE/, which means the app can't see the packages we've installed in the system's site-packages through rpms.

Since we're using vendor libs and expecting things like MySQLdb to come from system packages, we don't want to use WSGIPythonHome.

Comment 7

[Jeremy Orem [:oremj]]

Did someone check in a fix? The site isn't 500ing for me.

Comment 8

[Shyam Mani [:fox2mike]]

Yeah, Dave fixed stuff and I commented out BASELINE from conf.d/wsgi.conf

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; thanks, guys.

Comment 10

[Jeremy Orem [:oremj]]

Appears BASELINE isn't the problem. It's been on for a while and I just uncommented it and the site isn't 500ing. I'm fine with commenting it out again, just wanted to narrow down the cause.