Closed Bug 864422 Opened 11 years ago Closed 11 years ago

firefoxflicks-dev.allizom.org is currently down, returning 500

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

task
Not set
major

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: bensternthal, Assigned: rbryce)

References

()

Details

(Whiteboard: [fromAutomation])

firefoxflicks-dev.allizom.org is currently down, returning 500 error.

We noticed this happening this morning and are thinking it's related to the issues with the engagement cluster. Filing here to track, if its a dupe please feel free to close.
Severity: normal → major
Assignee: server-ops → server-ops-webops
Component: Server Operations → Server Operations: Web Operations
QA Contact: shyam → nmaul
Didnt see anything in the logs pointing to a specific failure.  I kicked apache and the site came back.
Assignee: server-ops-webops → rbryce
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
looks good to me.. verifying.
Status: RESOLVED → VERIFIED
Reopening:

firefoxflicks-dev.allizom.org appears to be down again. https://mozillians-dev.allizom.org also appears to be effected
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Whiteboard: [fromAutomation]
this was caused by some changes that were made to system packages earlier this morning. one of the packages that was updated appeared to have the wrong permissions causing all django/playdoh apps return the below tracebacks. i have gone ahead an applied an update to the permissions on this library and everything appears to be functioning once again.


[Wed Apr 24 09:43:22 2013] [info] [client 10.8.81.216] mod_wsgi (pid=23703, process='firefoxflicks-dev-ssl', application='firefoxflicks-dev.allizom.org|'): Loading WSGI script '/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/wsgi/playdoh.wsgi'.
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216] mod_wsgi (pid=23703): Target WSGI script '/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/wsgi/playdoh.wsgi' cannot be loaded as Python module.
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216] mod_wsgi (pid=23703): Exception occurred processing WSGI script '/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/wsgi/playdoh.wsgi'.
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216] Traceback (most recent call last):
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/wsgi/playdoh.wsgi", line 28, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     import manage
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/manage.py", line 24, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     manage.setup_environ(__file__, more_pythonic=True)
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor/src/funfactory/funfactory/manage.py", line 77, in setup_environ
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     settings = import_mod_by_name(os.environ['DJANGO_SETTINGS_MODULE'])
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor/src/funfactory/funfactory/manage.py", line 123, in import_mod_by_name
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     thing = _dot_lookup(thing, comp, import_path)
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor/src/funfactory/funfactory/manage.py", line 131, in _dot_lookup
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     __import__(import_path)
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/flicks/settings/__init__.py", line 3, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     from .base import *
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/flicks/settings/base.py", line 6, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     from flicks.base.util import reverse_lazy
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/flicks/base/util.py", line 5, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     from django.core.urlresolvers import NoReverseMatch
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib64/python2.6/site-packages/newrelic-1.10.2.38/newrelic/api/import_hook.py", line 99, in find_module
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     __import__(fullname)
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor/lib/python/django/core/urlresolvers.py", line 13, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     from django.http import Http404
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor/lib/python/django/http/__init__.py", line 124, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     from django.utils import timezone
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor/lib/python/django/utils/timezone.py", line 11, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     import pytz
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/data/www/firefoxflicks-dev.allizom.org/firefox-flicks/vendor-local/lib/python/pytz/__init__.py", line 35, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     from pkg_resources import resource_stream
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2675, in <module>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     add_activation_listener(lambda dist: dist.activate())
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 662, in subscribe
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     callback(dist)
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2675, in <lambda>
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     add_activation_listener(lambda dist: dist.activate())
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2177, in activate
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     self.insert_on(path)
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2284, in insert_on
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     self.check_version_conflict()
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2323, in check_version_conflict
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     for modname in self._get_metadata('top_level.txt'):
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2171, in _get_metadata
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     for line in self.get_metadata_lines(name):
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1168, in get_metadata_lines
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     return yield_lines(self.get_metadata(name))
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1160, in get_metadata
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     return self._get(self._fn(self.egg_info,name))
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1275, in _get
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216]     stream = open(path, 'rb')
[Wed Apr 24 09:43:22 2013] [error] [client 10.8.81.216] IOError: [Errno 13] Permission denied: '/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.3-py2.6.egg-info/top_level.txt'
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
Looks great to me, mozillians-dev.allizom and firefoxflicks-dev are back online. Thank you
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.