641880 - [input] OperationalError: (2003, "Can't connect to MySQL server on 'db-a01-ro' (111)")

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Matt Brandt [:mbrandt]]

I'm unable to reproduce this.

========================

Traceback (most recent call last):

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/django/input.mozilla.com/reporter/apps/input/decorators.py", line 125, in negotiated
    return beta(request, *args, **kwargs)

  File "/data/www/django/input.mozilla.com/reporter/apps/input/decorators.py", line 102, in wrapped
    return f(request, *args, **kwargs)

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/utils/decorators.py", line 76, in _wrapped_view
    response = view_func(request, *args, **kwargs)

  File "/data/www/django/input.mozilla.com/reporter/apps/input/decorators.py", line 43, in cached_view
    return f(request, *args, **kwargs)

  File "/data/www/django/input.mozilla.com/reporter/apps/dashboard/views.py", line 86, in beta
    return jingo.render(request, template, data)

  File "/data/www/django/input.mozilla.com/reporter/vendor/src/jingo/jingo/__init__.py", line 78, in render
    rendered = render_to_string(request, template, context)

  File "/data/www/django/input.mozilla.com/reporter/vendor/src/jingo/jingo/__init__.py", line 96, in render_to_string
    return template.render(**get_context())

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/jinja2/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/django/input.mozilla.com/reporter/apps/dashboard/templates/dashboard/beta.html", line 1, in top-level template code
    {% extends "base_3col.html" %}

  File "/data/www/django/input.mozilla.com/reporter/templates/base_3col.html", line 1, in top-level template code
    {% extends "base.html" %}

  File "/data/www/django/input.mozilla.com/reporter/templates/base.html", line 65, in top-level template code
    {% block content %}{% endblock %}

  File "/data/www/django/input.mozilla.com/reporter/templates/base_3col.html", line 10, in block "content"
    {% block col2 %}

  File "/data/www/django/input.mozilla.com/reporter/apps/dashboard/templates/dashboard/beta.html", line 21, in block "col2"
    {{ message_list(opinions) }}

  File "/data/www/django/input.mozilla.com/reporter/vendor/src/jingo/jingo/__init__.py", line 146, in wrapper
    t = env.get_template(template).render(context)

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/jinja2/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)

  File "/data/www/django/input.mozilla.com/reporter/apps/dashboard/templates/dashboard/message_list.html", line 1, in top-level template code
    {% if opinions %}

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/models/query.py", line 112, in __nonzero__
    iter(self).next()

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/models/query.py", line 106, in _result_iter
    self._fill_cache()

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/models/query.py", line 760, in _fill_cache
    self._result_cache.append(self._iter.next())

  File "/data/www/django/input.mozilla.com/reporter/vendor/src/django-cache-machine/caching/base.py", line 127, in __iter__
    obj = iterator.next()

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/models/query.py", line 269, in iterator
    for row in compiler.results_iter():

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/models/sql/compiler.py", line 672, in results_iter
    for rows in self.execute_sql(MULTI):

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/models/sql/compiler.py", line 726, in execute_sql
    cursor = self.connection.cursor()

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/backends/__init__.py", line 75, in cursor
    cursor = self._cursor()

  File "/data/www/django/input.mozilla.com/reporter/vendor/packages/Django/django/db/backends/mysql/base.py", line 297, in _cursor
    self.connection = Database.connect(**kwargs)

  File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)

  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 188, in __init__
    super(Connection, self).__init__(*args, **kwargs2)

OperationalError: (2003, "Can't connect to MySQL server on 'db-a01-ro' (111)")


<WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CSRF_COOKIE': 'be6f17973b178dda627012d1611aa1f6',
 'DOCUMENT_ROOT': '/data/www/django/input.mozilla.com/reporter',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'input.mozilla.com',
 'HTTP_USER_AGENT': 'check_http/v1.4.14 (nagios-plugins 1.4.14)',
 'PATH_INFO': u'/en-US/beta/',
 'PATH_TRANSLATED': '/data/www/django/input.mozilla.com/reporter/wsgi/reporter.wsgi/en-US/beta/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '10.8.75.9',
 'REMOTE_PORT': '43759',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/en-US/beta/',
 'SCRIPT_FILENAME': '/data/www/django/input.mozilla.com/reporter/wsgi/reporter.wsgi',
 'SCRIPT_NAME': u'',
 'SERVER_ADDR': '10.8.81.19',
 'SERVER_ADMIN': '[e-mail]',
 'SERVER_NAME': 'input.mozilla.com',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '',
 'SERVER_SOFTWARE': 'Apache',
 'mod_wsgi.application_group': 'input.mozilla.com|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'input',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0x2b41f15b9cb0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x2b41f2854d50>,
 'wsgi.input': <mod_wsgi.Input object at 0x2b41f1a47eb0>,
 'wsgi.loaded': datetime.datetime(2011, 3, 15, 9, 46, 26, 456228),
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

Comment 1

[Matt Brandt [:mbrandt]]

Reopening, a fresh traceback hot off the press.

/ga-IE/feedback

Traceback (most recent call last):

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/core/handlers/base.py", line 89, in get_response
   response = middleware_method(request)

 File "/data/www/django/input.mozilla.com/reporter/apps/input/middleware.py", line 76, in process_request
   settings.CACHE_COUNT_TIMEOUT)

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django-cache-machine/caching/base.py", line 265, in cached
   val = function()

 File "/data/www/django/input.mozilla.com/reporter/apps/input/middleware.py", line 74, in <lambda>
   site = cached(lambda: Site.objects.get(domain=domain),

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/models/manager.py", line 132, in get
   return self.get_query_set().get(*args, **kwargs)

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/models/query.py", line 344, in get
   num = len(clone)

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/models/query.py", line 82, in __len__
   self._result_cache = list(self.iterator())

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/models/query.py", line 273, in iterator
   for row in compiler.results_iter():

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/models/sql/compiler.py", line 680, in results_iter
   for rows in self.execute_sql(MULTI):

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/models/sql/compiler.py", line 734, in execute_sql
   cursor = self.connection.cursor()

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/backends/__init__.py", line 252, in cursor
   cursor = util.CursorWrapper(self._cursor(), self)

 File "/data/www/django/input.mozilla.com/reporter/vendor/src/django/django/db/backends/mysql/base.py", line 322, in _cursor
   self.connection = Database.connect(**kwargs)

 File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
   return Connection(*args, **kwargs)

 File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 188, in __init__
   super(Connection, self).__init__(*args, **kwargs2)

OperationalError: (2003, "Can't connect to MySQL server on 'db-a01-ro' (111)")

Comment 2

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Do you have the date from the TB email?

This line:  'wsgi.loaded': datetime.datetime(2011, 3, 15, 9, 46, 26, 456228),

That'll help Jason diagnose.

Comment 3

[Matt Brandt [:mbrandt]]

Thx dd. Jason here are the timestamps for the most recent TBs:

'wsgi.loaded': datetime.datetime(2011, 12, 13, 10, 4, 25, 884967),
'wsgi.loaded': datetime.datetime(2011, 12, 13, 7, 6, 53, 121311),

Comment 4

[Corey Shields [:cshields]]

Hi guys,

Pretty sure this was due to the phx1 zeus issues we had last week, which prevented DB vips from being reached.  We've fixed this, so going to close this for now but please reopen or file a new bug if it happens again.