902648 - [mozillians-dev] update configuration and kick it for playdoh upgrade

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Andrei Hajdukewycz [:sancus]]

We've upgraded to playdoh in https://github.com/mozilla/mozillians/commit/c786a8025aa1b71f809bf76de24a3687d4e4fdb5 but it seems like dev hasn't picked up the change, likely because the update is failing at some point.

We will need the same configuration changes as as Bug 872797, mainly moving the settings file and removing `from settings import *` as well as deleting the apps and settings directories that are leftover from the checkout.

If you find some additional problems not mentioned in that bug, ping in this one and we'll see if we can figure out what's going on.

Comment 1

[Justin Crawford [:hoosteeno] [:jcrawford]]

Just a note that this will block deploys to mozillians until resolved.

Comment 2

[Jason Crowe [:jd]]

Two issues, first there is no ' $PWD/vendor-local/src/django' directory, there are several with a django prefix but none with this name.

Next when attempting to deploy after doing the work in the other bug i am getting the following error:

[2013-08-07 15:06:41] [localhost] failed: cd /data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians && python ./manage.py migrate --list (0.073s)
[localhost] err: Traceback (most recent call last):
[localhost] err: File "./manage.py", line 20, in <module>
[localhost] err: manage.setup_environ(__file__, more_pythonic=True)
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 77, in setup_environ
[localhost] err: settings = import_mod_by_name(os.environ['DJANGO_SETTINGS_MODULE'])
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 123, in import_mod_by_name
[localhost] err: thing = _dot_lookup(thing, comp, import_path)
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 131, in _dot_lookup
[localhost] err: __import__(import_path)
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/mozillians/settings/__init__.py", line 7, in <module>
[localhost] err: from .local import *
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/mozillians/settings/local.py", line 7, in <module>
[localhost] err: DOMAIN = socket.gethostname()
[localhost] err: NameError: name 'socket' is not defined

Comment 3

[Andrei Hajdukewycz [:sancus]]

We don't have django installed in vendor-local, so that first thing is not necessary to repeat for us, unlike bedrock.

The second thing seems to be using the socket library to determine the domain, so you just need to add 'import socket' to the top of the settings file.

Comment 4

[Jason Crowe [:jd]]

Cool, now we have a new one :)

[2013-08-07 15:34:32] [localhost] failed: cd /data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians && python ./manage.py migrate --list (0.115s)
[localhost] err: Traceback (most recent call last):
[localhost] err: File "./manage.py", line 20, in <module>
[localhost] err: manage.setup_environ(__file__, more_pythonic=True)
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 77, in setup_environ
[localhost] err: settings = import_mod_by_name(os.environ['DJANGO_SETTINGS_MODULE'])
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 123, in import_mod_by_name
[localhost] err: thing = _dot_lookup(thing, comp, import_path)
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/vendor/src/funfactory/funfactory/manage.py", line 131, in _dot_lookup
[localhost] err: __import__(import_path)
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/mozillians/settings/__init__.py", line 7, in <module>
[localhost] err: from .local import *
[localhost] err: File "/data/genericrhel6-dev/src/mozillians-dev.allizom.org/mozillians/mozillians/settings/local.py", line 33, in <module>
[localhost] err: CACHES['default']['LOCATION'] = 'memcache-generic01:11211;memcache-generic02:11211'
[localhost] err: NameError: name 'CACHES' is not defined

Comment 5

[Andrei Hajdukewycz [:sancus]]

Since the import order has changed, things can't rely on stuff defined in base.py, so you need to define the full CACHES dict which is:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': 'memcache-generic01:11211;memcache-generic02:11211',
    }
}

Comment 6

[Jason Crowe [:jd]]

Soo, it sort of worked, there was a socket timeout which caused the deploy script to bork and it now thinks things are up to date. The site is WSOD with

[Wed Aug 07 16:08:35 2013] [error] [client 10.8.81.216] ImportError: No module named happyforms

traceback available upon request.

I will manually sync some things to make sure this is not the issue. FYI in future these long-running jobs should be &ed to avoid this socket timeout (this in the chief script you manage)

:)

Comment 7

[Andrei Hajdukewycz [:sancus]]

The virtualenv running the site needs the package dependencies for current playdoh, which you can install via pip install -r requirements/prod.txt ... at least that's how we install them. I'm not sure if there are any special procedures for webops installing these packages...

The required packages are in /requirements/prod.txt and that file additionally inherits the ones from /vendor/src/funfactory/funfactory/requirements/prod.txt

Comment 8

[Andrei Hajdukewycz [:sancus]]

Thanks for the pointer about the chief script.

Comment 9

[Jason Crowe [:jd]]

Umm, this is not really how things work on this cluster. We are not allowed to use pip (and never have been able to). Generally all dependencies must be reviewed for security concerns by the developers and then shipped in vendor-local. There are a very few exceptions where we have installed RPM packages but as this is a shared cluster that sort of thing tends to break other sites. Also we are not running virtualenvs on this cluster (and never have). We have discussed the possibility of running virtualenvs in the future or hosting our own pypi mirror where we can place validated packages, but alas this is not the state of things today.

I am really not sure how to proceed from here, I am juggling a bunch of things today and don’t really have time to work through, what at first blush appears to be, a rather long list of dependencies.

Do you have any ideas?

Comment 10

[Andrei Hajdukewycz [:sancus]]

happyforms added in https://github.com/mozilla/mozillians/commit/07133c4be564f1a941f9793f73e241a6aceebfe2 so that should fix this I hope. I couldn't find any other missing dependencies, but my install is probably not the perfect duplicate of dev.

Comment 11

[Andrei Hajdukewycz [:sancus]]

We had to fiddle with the middleware classes and INSTALLED_APPS and then set SITE_URL on dev's local.py

The site is now loading, but there may be some issues remaining(elasticsearch indexing may be timing out). Will open other bugs for this stuff if necessary.

Thanks for the help, jd!