949068 - moztrap-dev is down/returning a 500

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Screen Shot 2013-12-11 at 10.47.51 AM.png

https://moztrap-dev.allizom.org/ is returning a 500

Comment 1

[Brandon Burton [:solarce]]

(In reply to Stephen Donner [:stephend] from comment #0)
> Created attachment 8346048 [details]
> Screen Shot 2013-12-11 at 10.47.51 AM.png
> 
> https://moztrap-dev.allizom.org/ is returning a 500

It's currently failing with the following error

[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216] mod_wsgi (pid=32565): Exception occurred processing WSGI script '/data/www/moztrap-dev.allizom.org/moz-trap/moztrap/deploy/vendor_wsgi.py'.
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216] Traceback (most recent call last):
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/core/handlers/wsgi.py", line 219, in __call__
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     self.load_middleware()
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/core/handlers/base.py", line 45, in load_middleware
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     mod = import_module(mw_module)
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/utils/importlib.py", line 35, in import_module
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     __import__(name)
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/moztrap/view/users/middleware.py", line 10, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from . import views
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/moztrap/view/users/views.py", line 22, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from . import forms
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/moztrap/view/users/forms.py", line 12, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     import floppyforms as forms
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/floppyforms/__init__.py", line 16, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from floppyforms import gis
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/floppyforms/gis/__init__.py", line 1, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from floppyforms.gis.fields import *
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/floppyforms/gis/fields.py", line 2, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from django.contrib.gis.forms import GeometryField as BaseGeometryField
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/forms/__init__.py", line 2, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from django.contrib.gis.forms.fields import GeometryField
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/forms/fields.py", line 6, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from django.contrib.gis.geos import GEOSGeometry
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/geos/__init__.py", line 6, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from django.contrib.gis.geos.geometry import GEOSGeometry, wkt_regex, hex_regex
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/geos/geometry.py", line 14, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from django.contrib.gis.geos.coordseq import GEOSCoordSeq
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/geos/coordseq.py", line 9, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     from django.contrib.gis.geos.libgeos import CS_PTR
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/geos/libgeos.py", line 119, in <module>
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     _verinfo = geos_version_info()
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]   File "/data/www/moztrap-dev.allizom.org/moz-trap/requirements/vendor/lib/python/django/contrib/gis/geos/libgeos.py", line 115, in geos_version_info
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216]     if not m: raise GEOSException('Could not parse version info string "%s"' % ver)
[Wed Dec 11 11:46:17 2013] [error] [client 10.8.81.216] GEOSException: Could not parse version info string "3.4.2-CAPI-1.8.2 r3921"
[Wed Dec 11 11:46:17 2013] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

Comment 2

[Daniel Maher [:phrawzty]]

Was there a recent push or change that preceded this issue ?  Does that error output help pinpoint the issue at all ?

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Daniel Maher [:phrawzty] from comment #2)
> Was there a recent push or change that preceded this issue ?  Does that
> error output help pinpoint the issue at all ?

Dev pulls automatically based on commits to the GitHub repo: https://github.com/mozilla/moztrap/commits/master

Last commit was 6 days ago, and was this: https://github.com/mozilla/moztrap/commit/5801f7f4e4148dbcaa83a7474d72e1916e5a5d29

Cameron or someone else in IT/Web Ops/Webdev would be better to help analyze the error output.

Comment 5

[Cameron Dawson [:camd]]

I think (unless it's changed) that dev pulls from a branch called ``dev`` in the moztrap repo.  but that branch doesn't have any checkins since 9/30.  So I don't *think* a checkin could be at fault here.  

It looks like it's complainging about GeoDjango.  And I see reports of this error on Stack Overflow: http://stackoverflow.com/questions/18643998/geodjango-eosexception-error

However, MozTrap doesn't use Geo Django and it shouldn't be installed in the virtualenv for it.  Did anything change on the server side that installs that across different installations?

Comment 6

[Brandon Burton [:solarce]]

(In reply to Cameron Dawson [:camd] from comment #5)
> I think (unless it's changed) that dev pulls from a branch called ``dev`` in
> the moztrap repo.  but that branch doesn't have any checkins since 9/30.  So
> I don't *think* a checkin could be at fault here.  
> 
> It looks like it's complainging about GeoDjango.  And I see reports of this
> error on Stack Overflow:
> http://stackoverflow.com/questions/18643998/geodjango-eosexception-error
> 
> However, MozTrap doesn't use Geo Django and it shouldn't be installed in the
> virtualenv for it.  Did anything change on the server side that installs
> that across different installations?

Aha! So as part of some OS package upgrades GEOS got upgraded, "Dec 11 08:17:57 Updated: geos-3.4.2-1.rhel6.x86_64"

It looks like your vendored Django includes some gis stuff, requirements/vendor/lib/python/django/contrib/gis , and that's causing it to barf

I've rolled back geos to 3.3.2 for now, so the site loads, but I think for longterm you should upgrade the vendored Django to Django 1.4.8 or 1.5.4 (or the latest stable for the version you're tracking), is that feasible?

Comment 7

[Cameron Dawson [:camd]]

Brandon--  Yeah, definitely.  We have a branch where the upgrade to django 1.5 is in progress.  Though it's been on hold for a bit.  It is, however, on the top of the list for when dev resources can be put on it.

https://www.pivotaltracker.com/story/show/56641282

Comment 8

[Brandon Burton [:solarce]]

(In reply to Cameron Dawson [:camd] from comment #7)
> Brandon--  Yeah, definitely.  We have a branch where the upgrade to django
> 1.5 is in progress.  Though it's been on hold for a bit.  It is, however, on
> the top of the list for when dev resources can be put on it.
> 
> https://www.pivotaltracker.com/story/show/56641282

Ok, I'll pin the version in Puppet for now and RF this