Closed Bug 865628 Opened 11 years ago Closed 11 years ago

Status page failed when db was down

Categories

(Marketplace Graveyard :: Payments/Refunds, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
2013-05-16

People

(Reporter: kumar, Assigned: andy+bugzilla)

References

Details

http://sentry.stage.solitude.allizomaws.com/payments-stage/payments-stage/group/18/

This seemed to be a transient error but do we know why?

/services/status/: OperationalError (2005, "Unknown MySQL server host 'db1.stage.solitude.allizomaws.com' (0)")
  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/tastypie/resources.py", line 192, in wrapper
    response = callback(request, *args, **kwargs)

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/tastypie/resources.py", line 397, in dispatch_list
    return self.dispatch('list', request, **kwargs)

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/tastypie/resources.py", line 427, in dispatch
    response = method(request, **kwargs)

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/tastypie/resources.py", line 1029, in get_list
    objects = self.obj_get_list(request=request, **self.remove_api_resource_names(kwargs))

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/tastypie_services/services.py", line 150, in obj_get_list
    return [self.obj_get(request, **kwargs)]

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/tastypie_services/services.py", line 145, in obj_get
    if not obj.test():

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/solitude/lib/services/resources.py", line 63, in test
    self.test_db()

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/solitude/lib/services/resources.py", line 30, in test_db
    Seller.objects.exists()

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/django/db/models/manager.py", line 200, in exists
    return self.get_query_set().exists(*args, **kwargs)

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/django/db/models/query.py", line 565, in exists
    return self.query.has_results(using=self.db)

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/django/db/models/sql/query.py", line 441, in has_results
    return bool(compiler.execute_sql(SINGLE))

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 817, in execute_sql
    cursor = self.connection.cursor()

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/django/db/backends/__init__.py", line 319, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)

  File "/data/solitude/www/payments.allizom.org/releases/1366714299-origin.mas/venv/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 387, in _cursor
    self.connection = Database.connect(**kwargs)

  File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)

  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
Assignee: nobody → jthomas
I think this could be a issue with route53 or the ec2 instances solitude-stage is on. The stage instances have been up for 30+ days so to rule out it being a ec2 related issue I have rebuilt all the stage instances.
Was this a transient bug?  Can we close this?
It seems like this stopped after rebuilding the instances. Please reopen if it continues.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Status page shouldn't be spewing errors to sentry, but catching that and reporting the problem so nagios can deal with it.
Assignee: jthomas → amckay
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: /services/status/: Unknown MySQL server host db1.stage.solitude.allizomaws.com → Status page failed when db was down
Target Milestone: --- → 2013-05-16
https://github.com/mozilla/solitude/commit/39fe40
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.