701397 - [treestatus] Set up treestatus-dev

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Chris AtLee [:catlee]]

Can we get the treestatus app set up in the tbpl-dev environment so that it's accessible here:

tbpl-dev.allizom.org/status

The treestatus code is at https://github.com/catlee/treestatus

It requires a database and optionally memcached.

Only POST/PUT/DELETE methods require authentication. All GET requests are ok to pass though without auth.

Comment 1

[Corey Shields [:cshields]]

Brian, tbpl is hosted on the new generic cluster in phx1 (currently just dev and prod right now).  

Should be an easy addition to the code base, update script, and apache config.

Comment 2

[Brian Hourigan [:digi]]

Chris,

I see that the treestatus app has some pip dependencies, could you stick these into a vendor lib instead?

Comment 3

[Chris AtLee [:catlee]]

(In reply to Brian Hourigan [:digi] from comment #2)
> Chris,
> 
> I see that the treestatus app has some pip dependencies, could you stick
> these into a vendor lib instead?

How should this work? Create a 'vendor' subdirectory with everything inside it?

Comment 4

[Brian Hourigan [:digi]]

Chris:

So sorry for the delay, I was busy with some Q4 objectives and this fell off of my radar. You're right - the vendor directory will essentially include all of your pure Python dependencies.

The Mozillians project is a great example:

https://github.com/mozilla/mozillians

The vendor directory is a link to another repo which maintains all dependencies for the app. You can find more details on how to create the vendor library here:

http://jbalogh.github.com/zamboni/topics/packages/
http://playdoh.readthedocs.org/en/latest/packages.html#packages

I've already deployed python-memcached and python-sqlalchemy0.7.3 as these are not pure Python dependencies.

Comment 5

[Chris AtLee [:catlee]]

Ok, I've created a vendor submodule here:
https://github.com/catlee/treestatus-lib

Comment 6

[Chris AtLee [:catlee]]

ping - what's the status here? do I need to fix up the requirements / libraries at all?

Comment 7

[Brandon Burton [:solarce]]

Per a meeting on wed with arr, bear, dustin, coop, etc, I'll start working on this next week, once aus4-admin/balrog is functional.

Ping me on irc if you have questions about how things will work

Comment 8

[Brandon Burton [:solarce]]

I started work on this and it looks like vendor needs simplejson

[[e-mail]1 treestatus]# python testapp.py                                                                           │                                                                                                                                      
Traceback (most recent call last):                                                                                                     │                                                                                                                                        File "testapp.py", line 3, in <module>                                                                                               │                                                                                                                                      
    import treestatus.app, treestatus.model as model                                                                                   │                                                                                                                                        File "/data/genericrhel6-dev/src/tbpl-dev.allizom.org/treestatus/treestatus/app.py", line 9, in <module>                             │                                                                                                                                      
    from simplejson import dumps, loads                                                                                                │                                                                                                                                      ImportError: No module named simplejson

Can you add that I'll do a git pull on treestatus-lib and test again?

Thanks!

Comment 9

[Chris AtLee [:catlee]]

simplejson is listed in requirements/compiled.txt. Should I be putting compiled binaries into treestatus-lib?

Comment 10

[Brandon Burton [:solarce]]

Whoops, I missed it was in compiled, we have it in yum. I'll get it from there :)

Thanks!

Comment 11

[Brandon Burton [:solarce]]

I've been reviewing development.ini and the README

I opened bug 737669 to get a MySQL database setup.

Usually a .wsgi file is shipped with the app, to point mod_wsgi at, e.g. https://github.com/mozilla/mozillians/blob/master/wsgi/playdoh.wsgi

Where would I find that file for treestatus? I currently see the following

[[e-mail]1 treestatus]# find . -type f -name *wsgi*
./vendor/lib/python/paste/wsgiwrappers.py
./vendor/lib/python/paste/wsgilib.py
./vendor/lib/python/werkzeug/testsuite/wsgi.py
./vendor/lib/python/werkzeug/wsgi.py

Would the server:main settings still apply under mod_wsgi?

[server:main]
use = egg:Paste#http
host = 127.0.0.1
port = 5000

What about the filter-with line?

filter-with = who

Finally, does testing with `python testapp.py` make sense in this deployment scenario, because I'm seeing the following

[[e-mail]1 treestatus]# python testapp.py 
Traceback (most recent call last):
  File "testapp.py", line 3, in <module>
    import treestatus.app, treestatus.model as model
  File "/data/genericrhel6-dev/src/tbpl-dev.allizom.org/treestatus/treestatus/app.py", line 11, in <module>
    from repoze.who.config import make_middleware_with_config
  File "/data/genericrhel6-dev/src/tbpl-dev.allizom.org/treestatus/vendor/lib/python/repoze/who/config.py", line 11, in <module>
    from repoze.who.api import APIFactory
  File "/data/genericrhel6-dev/src/tbpl-dev.allizom.org/treestatus/vendor/lib/python/repoze/who/api.py", line 1, in <module>
    from zope.interface import implements
ImportError: No module named zope.interface

Comment 12

[Chris AtLee [:catlee]]

I added a .wsgi file, and updated the vendor library to include zope.interface

For deployment treestatus.wsgi and who.ini will need to be edited to have correct values for the db, public url, token secrets.

Comment 13

[Brandon Burton [:solarce]]

I am making progress. 

A couple things to update on.

1. Since tbpl.mozilla.org is a PHP app, I've had to setup treestatus as it's own site, since it is a python app, so I'm setting it up as treestatus.mozilla.org , but, we can make tpbl.mozilla.org/status a 302 to the subdomain (I'm starting with treestatus-dev.allizom.org, but assuming it'll eventually roll forward into production)
2. The dev site is setup, but I'm still working through dependency issues.

a. since zope.interface is a compiled one, I'll need to see about getting this into our yum repo
b. Right now I'm blocked on the following error, which may just be a path issue. I found two other minor path changes, I'm about to do a pull request on.

[Thu Mar 22 15:56:20 2012] [info] [client 10.8.32.5] mod_wsgi (pid=30721, process='treestatus-dev', application='treestatus-dev.allizom.org|'): Loading WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5] mod_wsgi (pid=30721): Target WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi' cannot be loaded as Python module.
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5] mod_wsgi (pid=30721): Exception occurred processing WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5] Traceback (most recent call last):
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi", line 8, in <module>
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5]     import treestatus.app, treestatus.model as model
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus/app.py", line 11, in <module>
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5]     from repoze.who.config import make_middleware_with_config
[Thu Mar 22 15:56:21 2012] [error] [client 10.8.32.5] ImportError: No module named repoze.who.config
[Thu Mar 22 15:56:21 2012] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

Comment 14

[Brandon Burton [:solarce]]

Unfortunately, I'm still stuck on this error, I've tried fiddling with the import path as best I know how and it doesn't seem to be finding this. I'm not sure if it is finding other things in vendor, though I think it might be.

Is there some more debug logging you could add that would help you track down what's up with this import error?

I can give you shell access on one of the web servers and I'm just testing with curl -H "Host: treestatus-dev.allizom.org" -v http://generic2.dev.seamicro.phx1.mozilla.com:81/ , let me know

Apache error log:

[Mon Mar 26 08:44:59 2012] [info] [client 10.8.32.5] mod_wsgi (pid=28582, process='treestatus-dev', application='treestatus-dev.allizom.org|'): Loading WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5] mod_wsgi (pid=28582): Target WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi' cannot be loaded as Python module.
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5] mod_wsgi (pid=28582): Exception occurred processing WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5] Traceback (most recent call last):
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi", line 9, in <module>
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5]     import treestatus.app, treestatus.model as model
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus/app.py", line 11, in <module>
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5]     from repoze.who.config import make_middleware_with_config
[Mon Mar 26 08:44:59 2012] [error] [client 10.8.32.5] ImportError: No module named repoze.who.config
[Mon Mar 26 08:44:59 2012] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

Comment 15

[Chris AtLee [:catlee]]

Can you try again with https://github.com/catlee/treestatus/commit/7f64c125581f3f981bf8e6c736dbc9f025796c64

Comment 16

[Brandon Burton [:solarce]]

Making progress

[Thu Mar 29 21:32:06 2012] [info] mod_wsgi (pid=15054): Create interpreter 'treestatus-dev.allizom.org|'.
[Thu Mar 29 21:32:06 2012] [info] [client 10.8.32.5] mod_wsgi (pid=15054, process='treestatus-dev', application='treestatus-dev.allizom.org|'): Loading WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5] mod_wsgi (pid=15054): Target WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi' cannot be loaded as Python module.
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5] mod_wsgi (pid=15054): Exception occurred processing WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5] Traceback (most recent call last):
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi", line 16, in <module>
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5]     'debug': False,
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus/app.py", line 536, in wsgiapp
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5]     app.wsgi_app = make_middleware_with_config(app.wsgi_app, config, config.get('who_config', 'who.ini'))
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/vendor/lib/python/repoze/who/config.py", line 184, in make_middleware_with_config
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5]     parser.parse(open(config_file))
[Thu Mar 29 21:32:08 2012] [error] [client 10.8.32.5] IOError: [Errno 2] No such file or directory: 'who.ini'
[Thu Mar 29 21:32:08 2012] [debug] mod_headers.c(768): headers: ap_headers_error_filter()

I'm gonna poke at it a bit tonight

Comment 17

[Brandon Burton [:solarce]]

Got the who.ini path sorted with a minor tweak. See https://github.com/catlee/treestatus/pull/3

Sadly, that led to the next error, which appears to be a SQL error

[Fri Mar 30 00:29:31 2012] [info] mod_wsgi (pid=5807): Attach interpreter ''.
[Fri Mar 30 00:29:32 2012] [info] mod_wsgi (pid=5802): Attach interpreter ''.
[Fri Mar 30 00:29:32 2012] [info] mod_wsgi (pid=5794): Attach interpreter ''.
[Fri Mar 30 00:29:32 2012] [info] mod_wsgi (pid=5807): Create interpreter 'treestatus-dev.allizom.org|'.
[Fri Mar 30 00:29:32 2012] [info] [client 10.8.32.5] mod_wsgi (pid=5807, process='treestatus-dev', application='treestatus-dev.allizom.org|'): Loading WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Fri Mar 30 00:29:33 2012] [info] mod_wsgi (pid=5803): Attach interpreter ''.
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5] mod_wsgi (pid=5807): Target WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi' cannot be loaded as Python module.
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5] mod_wsgi (pid=5807): Exception occurred processing WSGI script '/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi'.
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5] Traceback (most recent call last):
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi", line 20, in <module>
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     model.DbBase.metadata.create_all()
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/schema.py", line 1786, in create_all
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     bind.create(self, checkfirst=checkfirst, tables=tables)
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1129, in create
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     self._run_visitor(self.dialect.schemagenerator, entity, connection=connection, **kwargs)
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1158, in _run_visitor
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     visitorcallable(self.dialect, conn, **kwargs).traverse(element)
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 89, in traverse
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     return traverse(obj, self.__traverse_options__, self._visitor_dict)
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 200, in traverse
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     return traverse_using(iterate(obj, opts), obj, visitors)
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 194, in traverse_using
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]     meth(target)
[Fri Mar 30 00:29:43 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/compiler.py", line 831, in visit_metadata
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     self.traverse_single(table)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/visitors.py", line 79, in traverse_single
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     return meth(obj)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/sql/compiler.py", line 870, in visit_table
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     self.execute()
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1812, in execute
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     return self.connection.execute(self.buffer.getvalue())
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 824, in execute
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     return Connection.executors[c](self, object, multiparams, params)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 888, in _execute_text
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     return self.__execute_context(context)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 896, in __execute_context
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 950, in _cursor_execute
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     self._handle_dbapi_exception(e, statement, parameters, cursor, context)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5]     raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)
[Fri Mar 30 00:29:44 2012] [error] [client 10.8.32.5] ProgrammingError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NOT NULL, \\n\\ttoken VARCHAR(100) NOT NULL, \\n\\tPRIMARY KEY (who)\\n)' at line 2") '\\nCREATE TABLE tokens (\\n\\twho VARCHAR NOT NULL, \\n\\ttoken VARCHAR(100) NOT NULL, \\n\\tPRIMARY KEY (who)\\n)\\n\\n' ()

Comment 18

[Chris AtLee [:catlee]]

ok, I've fixed up the schema to play nicely with mysql. can you give it a shot now?

Comment 19

[Brandon Burton [:solarce]]

It's alive!!!! https://treestatus-dev.allizom.org/

So now that this is alive. I'll wire up the 302 from tbpl-dev in Apache and get a cron job setup to auto-update -dev, I should have these done tomorrow

If you can test out the functionality on -dev, then we can get it rolling into a -stage and a -prod setup and go live :)

Comment 20

[Chris AtLee [:catlee]]

I'm just getting 500 errors from it right now.

Comment 21

[Brandon Burton [:solarce]]

How do I reproduce? https://treestatus-dev.allizom.org/help and https://treestatus-dev.allizom.org/ load fine for me

Comment 22

[Chris AtLee [:catlee]]

I get it about 80% of the time at https://treestatus-dev.allizom.org/.

Can I get access to the box to look at the logs?

Comment 23

[Brandon Burton [:solarce]]

I'm seeing it 100% now too :(

I can give you a shell, but the Apache logs are root owned and I'd have to ask about sudo since generic-dev is shared by a couple dozen sites

I'll check out the logs real quick

Comment 24

[Brandon Burton [:solarce]]

[[e-mail]1 treestatus.mozilla.org]# cat error_log_2012-04-02-21
[Mon Apr 02 14:35:33 2012] [error] ERROR:treestatus.app:Exception on / [GET]
[Mon Apr 02 14:35:33 2012] [error] Traceback (most recent call last):
[Mon Apr 02 14:35:33 2012] [error]   File "/data/www/treestatus-dev.allizom.org/tree-status/vendor/lib/python/flask/app.py", line 1292, in wsgi_app
[Mon Apr 02 14:35:33 2012] [error]     response = self.full_dispatch_request()
[Mon Apr 02 14:35:33 2012] [error]   File "/data/www/treestatus-dev.allizom.org/tree-status/vendor/lib/python/flask/app.py", line 1062, in full_dispatch_request
[Mon Apr 02 14:35:33 2012] [error]     rv = self.handle_user_exception(e)
[Mon Apr 02 14:35:33 2012] [error]   File "/data/www/treestatus-dev.allizom.org/tree-status/vendor/lib/python/flask/app.py", line 1060, in full_dispatch_request
[Mon Apr 02 14:35:33 2012] [error]     rv = self.dispatch_request()
[Mon Apr 02 14:35:33 2012] [error]   File "/data/www/treestatus-dev.allizom.org/tree-status/vendor/lib/python/flask/app.py", line 1047, in dispatch_request
[Mon Apr 02 14:35:33 2012] [error]     return self.view_functions[rule.endpoint](**req.view_args)
[Mon Apr 02 14:35:33 2012] [error]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus/app.py", line 303, in index
[Mon Apr 02 14:35:33 2012] [error]     trees = [t for t in status.get_trees().values()]
[Mon Apr 02 14:35:33 2012] [error]   File "/data/www/treestatus-dev.allizom.org/tree-status/treestatus/app.py", line 157, in get_trees
[Mon Apr 02 14:35:33 2012] [error]     for t in model.Session.query(model.DbTree):
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 1292, in __iter__
[Mon Apr 02 14:35:33 2012] [error]     return self._execute_and_instances(context)
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 1295, in _execute_and_instances
[Mon Apr 02 14:35:33 2012] [error]     result = self.session.execute(querycontext.statement, params=self._params, mapper=self._mapper_zero_or_none())
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 755, in execute
[Mon Apr 02 14:35:33 2012] [error]     clause, params or {})
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 824, in execute
[Mon Apr 02 14:35:33 2012] [error]     return Connection.executors[c](self, object, multiparams, params)
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 872, in _execute_clauseelement
[Mon Apr 02 14:35:33 2012] [error]     parameters=params
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 938, in __create_execution_context
[Mon Apr 02 14:35:33 2012] [error]     return dialect.execution_ctx_cls(dialect, connection=self, **kwargs)
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 170, in __init__
[Mon Apr 02 14:35:33 2012] [error]     self.cursor = self.create_cursor()
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 258, in create_cursor
[Mon Apr 02 14:35:33 2012] [error]     return self._connection.connection.cursor()
[Mon Apr 02 14:35:33 2012] [error]   File "/usr/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 576, in connection
[Mon Apr 02 14:35:33 2012] [error]     raise exc.InvalidRequestError("Can't reconnect until invalid transaction is rolled back")
[Mon Apr 02 14:35:33 2012] [error] InvalidRequestError: Can't reconnect until invalid transaction is rolled back
[Mon Apr 02 14:35:33 2012] [debug] mod_headers.c(743): headers: ap_headers_output_filter()

Comment 25

[Brandon Burton [:solarce]]

*sigh* that posted before I meant it to.

That's the error I'm seeing on every request I make.

I see one entry of the following before the error above started happening

[Mon Apr 02 06:32:39 2012] [error]     raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)
[Mon Apr 02 06:32:39 2012] [error] OperationalError: (OperationalError) (2006, 'MySQL server has gone away') u'SELECT trees.tree AS trees_tree, trees.status AS trees_status, trees.reason AS trees_reason \\nFROM trees \\nWHERE trees.tree = %s' [u'favicon.ico']

I'm going to talk to the dbas to look at the DB, but from an app point of view, is the app meant to deal with being on 2+ app servers and having connections to the app bounce between multiple app servers with each request?

Comment 26

[Brandon Burton [:solarce]]

*sigh* that posted before I meant it to.

That's the error I'm seeing on every request I make.

I see one entry of the following before the error above started happening

[Mon Apr 02 06:32:39 2012] [error]     raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)
[Mon Apr 02 06:32:39 2012] [error] OperationalError: (OperationalError) (2006, 'MySQL server has gone away') u'SELECT trees.tree AS trees_tree, trees.status AS trees_status, trees.reason AS trees_reason \\nFROM trees \\nWHERE trees.tree = %s' [u'favicon.ico']

I'm going to talk to the dbas to look at the DB, but from an app point of view, is the app meant to deal with being on 2+ app servers and having connections to the app bounce between multiple app servers with each request?

Comment 27

[Brandon Burton [:solarce]]

I opened bug 741596 to also have the state of the DB checked on, but if you can confirm about the multi server impact of things

Also, the following are the last to two lines of treestatus.wsgi

# Create any db tables we need
model.DbBase.metadata.create_all()

Would that mean this is executed on every HTTP request?

Comment 28

[Chris AtLee [:catlee]]

I bet the pool recycle time isn't getting set, I'll check in on that.

I don't believe that model.DbBase.metadata.create_all() gets called per request, only when the application is instantiated. Hopefully you're not creating the app fresh per request?

Comment 29

[Brandon Burton [:solarce]]

I'm not sure what you mean by instantiated.

I know that the mod_wsgi daemon will get restarted if Apache is restart, or when the .wsgi file is touched, which we usually have done every 15 minutes as part of the continuous updates of -dev sites, -stage and -prod are usually only done during manual pushes.

Does it sound like this would get called any time the mod_wsgi daemon is restarted?

Comment 30

[Chris AtLee [:catlee]]

I've pushed new changes that should fix the mysql error you refer to above. Either these changes aren't getting picked up, or now new errors are popping up because I'm still getting HTTP 500 errors.

Debugging via bugmail is hard.

Comment 31

[Brandon Burton [:solarce]]

SSH access given to genericrhel6-dev to look at logs and latest code pushed.

https://treestatus-dev.allizom.org/help and https://treestatus-dev.allizom.org/login are loading now

From irc it sounded like you made progress

How are things looking for you?

Comment 32

[Chris AtLee [:catlee]]

yes, no longer getting 500 errors. now I'm unable to login, which I suspect is due to browserid somewhere.

Comment 33

[Brandon Burton [:solarce]]

Ok, so is that a development fix? 

Are you at the point where you want this to auto-update every 15 minutes via cron?

If so, should it have more than just 'git pull' in the update script?

Comment 34

[Chris AtLee [:catlee]]

The auto-updating is working for some things, e.g. the html templates. However, it doesn't look like the new application code gets deployed. Does your auto-update script take care of restarting the wsgi daemons, or should my application kill itself off when new code is detected?

Comment 35

[Brandon Burton [:solarce]]

Yes, the script runs 'touch /data/www/treestatus-dev.allizom.org/tree-status/treestatus.wsgi' at the end, and I see mod_wsgi recycling, in the error_log_$date

So that should be good, what issue are you currently seeing?

Comment 36

[Chris AtLee [:catlee]]

Still having the problems where new application code doesn't get picked up.

Can you describe how new code gets deployed? Who is responsible for killing the old processes?

Comment 37

[Chris AtLee [:catlee]]

(In reply to Chris AtLee [:catlee] from comment #36)
> Still having the problems where new application code doesn't get picked up.
> 
> Can you describe how new code gets deployed? Who is responsible for killing
> the old processes?

Still having this issue. I pushed new code about an hour ago, and the application is still running the old code.

Looks like it's running on node202.seamicro.phx1.mozilla.com, pid 11244 which started at Wed May  9 18:56:48 2012

Comment 38

[Brandon Burton [:solarce]]

I am working :jabba on a permanent fix for this, but realistically it won't be in place until next week at the earliest, given SJC1 activity

In the mean time I added a cron job on -dev to touch the wsgi file directly, but I'm leaving this bug open to track testing our permanent solution (it dovetails with some Puppet ssh key stuff :jabba's been working on)

Thanks for your patience

Comment 39

[Brandon Burton [:solarce]]

Sorry for the delay, I've gotten caught up in a number of equally important bugs in the last couple of weeks and pulled into dealing with some tasks that weren't part of bugs, but post SJC1 cleanup/improvements that suddenly were needed more urgently. 

Unfortunately this has resulted in my losing state on this bug. I'm triaging this and other bugs today and will be asking another member of the webops team to try and make some forward progress on this bug this week, as I'll be on PTO getting LASIK tomorrow, Thursday, and potentially Friday.

Please ping me on irc if you have any questions for me specifically.

Comment 40

[Brandon Burton [:solarce]]

Renormalizing priority levels... P4 is "normal" now.

Comment 41

[Jake Maul [:jakem]]

Sorry for the very long delay on this... I believe this is fixed now. The problem is that Commander (issue-multi-command) does not accept a command on the command line when run via a dumb terminal (aka: cron). Thus the "touch" command doesn't actually work.

Instead, the solution is to pass the command to issue-multi-command on stdin. This works. There's a timing issue which stupidly causes the touch to happen before the code is actually deployed, but in the end this just causes a small delay until the next cron interval comes along and does the right thing. To compensate I've sped up the cron to run every 10 minutes (though we can fix the underlying problem if it becomes a hassle).


I don't see any other remaining work in this bug, so I'm going to close it out for now. If anything does pop up, please let us know. Thank you.

Comment 42

[Jake Maul [:jakem]]

Note that I do still see a 500 ISE shown at http://treestatus-dev.allizom.org/. The error from the logfile is:

[Wed Sep 19 13:37:13 2012] [error] OperationalError: (OperationalError) (1054, "Unknown column 'trees.message_of_the_day' in 'field list'") u'SELECT trees.tree AS trees_tree, trees.status AS trees_status, trees.reason AS trees_reason, trees.message_of_the_day AS trees_message_of_the_day \\nFROM trees' []

The update script does not currently seem to do anything with respect to DB schema changes. There is a "manage.py syncdb" command, but it's commented out (presumably only needed once). Is there a South or Schematic command it should be running?

Comment 43

[Chris AtLee [:catlee]]

There's no automatic schema migration right now. I guess we'll need that soon!