669398 - Need boxes for DLL directory

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Laura Thomson :laura]

We will aim to ship the DLL directory sometime in Q3 as a webdev goal.  Background for this is in bug 577613.

It will be a playdoh/MySQL app with LDAP auth for writes but publicly readable.  More information as it comes to hand, this bug is for planning purposes at present.

Comment 2

[Corey Shields [:cshields]]

can we talk in person next week about this?  I've got a lot of questions.  :)

Comment 3

[Laura Thomson :laura]

Yes, that sounds fine.  We'll chat in PDX!

Comment 4

[Corey Shields [:cshields]]

this will go on the generic cluster in phx1.. 

we'll help out jason in getting this setup. 

Laura, can you start a page for this (using the template in mana) under https://mana.mozilla.org/wiki/display/websites/Home

Comment 5

[Corey Shields [:cshields]]

Laura: ping?

Comment 6

[Brandon Savage [:brandon]]

A mana page exists for this: https://mana.mozilla.org/wiki/display/websites/Dragnet The project is called Dragnet.

Comment 7

[Jason Crowe [:jd]]

I will need at least the working primary domain name moving forward.  I can extrapolate the dev and stage domains from there.

Thanks

Comment 8

[Laura Thomson :laura]

dragnet.mozilla.org

Comment 9

[Laura Thomson :laura]

Could I get a status update here please?

Comment 10

[Jason Crowe [:jd]]

Laura,

Is there any repository for this?  The mana page mentions git but I do not have any URL.  I can probably get at least a dev environment up by the end of the week if I get the git info soonish.

Comment 11

[Laura Thomson :laura]

This is the basic repo
https://github.com/brandonsavage/dlldir

We should probably move it into the mozilla github, but that will do to begin with.

Comment 12

[Brandon Savage [:brandon]]

This has been moved into Mozilla's domain. https://github.com/mozilla/dragnet

Comment 13

[Jason Crowe [:jd]]

When attempting to deploy this site I see there is already a settings/local.py file in git.  I need this file removed from git and to have only the settings/local.py-dist file.

Also are you not using the new layout (like dragnet/dragnet/settings)?  This does not matter as long as we are careful, just checking.

Please let me know when the local.py file has been removed as I cannot continue until this is done.

Thanks

Comment 14

[Brandon Savage [:brandon]]

Fixed. Please use the Mozilla repo, as mine no longer exists.

Comment 15

[Jason Crowe [:jd]]

Hey,

Getting the following error when deploying the site:

ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.

Also what sort of LDAP access are you after?  Just the generic ro for auth or something more?

Comment 16

[Jason Crowe [:jd]]

BUMP

Sorry to pester but I cannot move forward without a fix.

Comment 17

[James Socol [:jsocol, :james]]

Jason, where are you seeing that, in the Apache error log? Can you paste any more of a traceback?

Comment 18

[Jason Crowe [:jd]]

James,

Yes, Apache error log. Traceback follows:

[Fri Feb 03 10:40:43 2012] [info] mod_wsgi (pid=3016): Create interpreter 'dragnet-dev.allizom.org|'.
[Fri Feb 03 10:40:43 2012] [info] [client 10.8.33.240] mod_wsgi (pid=3016, process='dragnet-dev-ssl', application='dragnet-dev.allizom.org|'): Loading WSGI script '/data/www/dragnet-dev.allizom.org/dragnet/wsgi/playdoh.wsgi'.
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240] mod_wsgi (pid=3016): Exception occurred processing WSGI script '/data/www/dragnet-dev.allizom.org/dragnet/wsgi/playdoh.wsgi'.
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240] Traceback (most recent call last):
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet-dev.allizom.org/dragnet/vendor/src/django/django/core/handlers/wsgi.py", line 250, in __call__
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]     self.load_middleware()
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet-dev.allizom.org/dragnet/vendor/src/django/django/core/handlers/base.py", line 39, in load_middleware
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]     for middleware_path in settings.MIDDLEWARE_CLASSES:
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet-dev.allizom.org/dragnet/vendor/src/django/django/utils/functional.py", line 276, in __getattr__
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]     self._setup()
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet-dev.allizom.org/dragnet/vendor/src/django/django/conf/__init__.py", line 40, in _setup
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240]     raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
[Fri Feb 03 10:40:47 2012] [error] [client 10.8.33.240] ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
[Fri Feb 03 10:40:47 2012] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

Comment 19

[Jason Crowe [:jd]]

Hi,

Any traction on this?

Thanks

Comment 20

[Laura Thomson :laura]

Just asked Brandon to take a look, sorry for the delay.

Comment 21

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Peter, Kumar?

Comment 22

[Fred Wenzel [:wenzel]]

Could that be caused by an import error caused by a typo in the config file?

manage.py has a habit of masking such things, or it used to anyway.

Comment 23

[James Socol [:jsocol, :james]]

(In reply to Fred Wenzel [:wenzel] from comment #22)
> Could that be caused by an import error caused by a typo in the config file?

Based on the stack trace, probably not. (Not willing to be definitive.)

It's coming from django.conf.LazySettings._setup()[1], which is specifically converting a KeyError into an ImportError.

[1] https://github.com/django/django/blob/73d6c3c5f3fd17b846219f7a5a23729f7b8f563b/django/conf/__init__.py#L36

Comment 24

[Brandon Savage [:brandon]]

Jason, Peterbe suggested adding the following:

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

This goes right under line 13. Let met know what happens.

Comment 25

[Peter Bengtsson [:peterbe]]

The "answer" is that the .wsgi file relies on manage.py settings DJANGO_SETTINGS_MODULE in the os.environ. It's hard to keep up with the nicely progressing funfactory/playdoh stuff. (In reply to Brandon Savage [:brandon] from comment #24)
> Jason, Peterbe suggested adding the following:
> 
> os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
> 
> This goes right under line 13. Let met know what happens.

Why don't you make the change, run it locally in apache or inside a vagrant and see if it works? I can help if you need it.

Comment 26

[Jason Crowe [:jd]]

FWIW adding that line seems to fix things enough to make the site load.  It is not persistent for me due to that file being in git.  Can someone update that file in git and then let me know?

Thanks

Comment 27

[Brandon Savage [:brandon]]

I've updated the Github repo with this fix.

Comment 28

[Jason Crowe [:jd]]

I set up both a stage and prod instance for this site, updated the mana page, and opened a bug for the prd SSL cert.

AFAIK this will finish all required set up for this site.  If there is anything missing please let me know.

Regards

Comment 29

[Brandon Savage [:brandon]]

It appears the staging site for this has internal server errors for the CSS and JS files. Any chance you can provide the logs for these errors?

Comment 30

[Jason Crowe [:jd]]

FYI, I have the SSL cert set up in prod.

@Brandon,

Here is the trace-back from the server logs.


[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240] mod_wsgi (pid=13077): Exception occurred processing WSGI script '/data/www/dragnet.allizom.org/dragnet/wsgi/playdoh.wsgi'.
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240] Traceback (most recent call last):
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/core/handlers/wsgi.py", line 272, in __call__
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     response = self.get_response(request)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/core/handlers/base.py", line 153, in get_response
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/core/handlers/base.py", line 218, in handle_uncaught_exception
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     return callback(request, **param_dict)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/utils/decorators.py", line 93, in _wrapped_view
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     response = view_func(request, *args, **kwargs)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/views/defaults.py", line 30, in server_error
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     t = loader.get_template(template_name) # You need to create a 500.html template.
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/template/loader.py", line 157, in get_template
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     template, origin = find_template(template_name)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/template/loader.py", line 134, in find_template
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     source, display_name = loader(name, dirs)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/django/django/template/loader.py", line 42, in __call__
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     return self.load_template(template_name, template_dirs)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/jingo/jingo/__init__.py", line 178, in load_template
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     template = env.get_template(template_name)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/data/www/dragnet.allizom.org/dragnet/vendor/src/jingo/jingo/__init__.py", line 29, in get_template
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     return super(Environment, self).get_template(name, parent, globals)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 716, in get_template
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     return self._load_template(name, self.make_globals(globals))
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 690, in _load_template
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     template = self.loader.load(self, name, globals)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/usr/lib/python2.6/site-packages/jinja2/loaders.py", line 115, in load
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     source, filename, uptodate = self.get_source(environment, name)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]   File "/usr/lib/python2.6/site-packages/jinja2/loaders.py", line 378, in get_source
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240]     raise TemplateNotFound(template)
[Wed Feb 15 08:50:55 2012] [error] [client 10.8.33.240] TemplateNotFound: 500.html

Regards

Comment 31

[Brandon Savage [:brandon]]

I didn't see the FQDNs for the stage and dev boxes in the Mana update; is there a way for me to SSH in and see the configuration? I think we're trying to have Django serve the assets, which wouldn't work well. I think I may be able to give you some solutions if I can take a look at the config.

Comment 32

[Jason Crowe [:jd]]

Attachment: local.py

Brandon,

Attached is the (redacted) local.py configuration file for stage.  This is the only configuration that is custom (everything else is in git).

Also with the obvious exception of the MySql settings and the DEBUG setting this file is identical in dev and prod

Comment 33

[Jason Crowe [:jd]]

Brandon,

Just checking to see if there has been any progress on this?

Thanks

Comment 34

[Brandon Savage [:brandon]]

Jason, I actually need the Apache configuration information for this domain. Here's the reason...

When deployed to stage or production, Django doesn't serve assets like CSS and JS files. We need to separately alias those files in Apache so that they can be served properly. The reason assets are served on dev but not stage is because in dev most developers run the manage.py built-in server, so we have designed Playdoh to serve those files through that server during development. But this is not the case for production.

So, for Apache's sake, /media needs to be aliased, probably to /data/www/dragnet.allizom.org/dragnet/media on stage and prod ONLY.

Comment 35

[Jason Crowe [:jd]]

Brandon,

I added an alias for the media directory to both stage and prod.  It looks like these are both working now.  This should finish this up so I will close this out now.  Feel free to reopen if further issues arise.

Regards
Jason

Comment 36

[Brandon Savage [:brandon]]

There appears to be some older submodules gumming up the works, namely in the forms department.

I would like you to run `git submodule update --init --recursive` and make sure that all the submodules are up to date on both dev and stage.

Comment 37

[Jason Crowe [:jd]]

Brandon,

I ran this command on both dev and stage.  There was not much in the way of output so I doubt the effectiveness.  I also manually ran an update on dev and stage to ensure there were no errors and that the auto updater on dev was not stuck, all appeared well.  Please let me know if you need me to do anything else.

Regards