1078814 - 500 Server Errors when editing test cases in MozTrap

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Marcia Knous [:marcia]]

Noticing this today when trying to edit existing test cases.

The remote test team has reported it as well.

Comment 1

[Peter Bylenga [:PBylenga]]

I personally hit it when attempting to logout of moztrap. 

Smoke team experienced silent failings of trying to pass/fail test cases (example: no success or error given, no change is made to the test case).

Comment 2

[Derek Harris [:DerekH] - Dont need info. Last day 6/5/15]

Test pass team is running into the same error, testing on this run: http://mzl.la/1re3s3X

When we try to pass, fail, invalidate, skip, or block a case, nothing appears to happen.

We are able to navigate pages, and filter searches just fine.

Comment 3

[Peter Bylenga [:PBylenga]]

Here's a link the the run we were experience the issues with: https://moztrap.mozilla.org/runtests/run/5569/env/27835/

Comment 4

[Derek Moore [:dmoore]]

Moztrap is considered a critical service, so raising priority accordingly.

Comment 5

[Chris Turra [:cturra]]

i have not been able to track down any error logs or tracebacks for these http 500 errors. the error message states: "our developers have been notified..." does this mean tracebacks are being emailed to the developers? if so, can we get one dumped into this bug?

Comment 6

[Chris Turra [:cturra]]

i managed to trace down a traceback by adding myself to the django ADMIN list for the site. the following is what is being reported. specifically: 

"DatabaseError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')"

i am going to poke the DBAs.



Traceback (most recent call last):

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/core/handlers/base.py", line 186, in get_response
    response = middleware_method(request, response)

  File "/usr/lib64/python2.6/site-packages/newrelic-2.22.1.20/newrelic/hooks/framework_django.py", line 321, in wrapper
    return wrapped(*args, **kwargs)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/contrib/sessions/middleware.py", line 36, in process_response
    request.session.save()

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/contrib/sessions/backends/cached_db.py", line 42, in save
    super(SessionStore, self).save(must_create)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/contrib/sessions/backends/db.py", line 58, in save
    obj.save(force_insert=must_create, using=using)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/db/models/query.py", line 1593, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/db/models/sql/compiler.py", line 912, in execute_sql
    cursor.execute(sql, params)

  File "/data/www/moztrap.mozilla.org/moz-trap/requirements/vendor/lib/python/django/db/backends/mysql/base.py", line 114, in execute
    return self.cursor.execute(query, args)

  File "/usr/lib64/python2.6/site-packages/newrelic-2.22.1.20/newrelic/hooks/database_dbapi2.py", line 22, in execute
    *args, **kwargs)

  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)

  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue

DatabaseError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')


<WSGIRequest
path:/users/login/,
GET:<QueryDict: {u'next': [u'/manage/cases/']}>,
POST:<QueryDict: {}>,
COOKIES:{'WT_FPC': 'id=63.245.220.240-2503254000.30218500:lv=1392795623402:ss=1392794403118',
 '__utma': '150903082.282625269.1351702627.1412370493.1412377808.303',
 '__utmx': '150903082.QhvDOg2zQqa9threH1A1kQ$71153379-29:1.SNKFJ4VqRziIW1TLqjhpAw$71153379-28:',
 '__utmxx': '150903082.QhvDOg2zQqa9threH1A1kQ$71153379-29:1383864788:15552000.SNKFJ4VqRziIW1TLqjhpAw$71153379-28:1398901249:15552000',
 '__utmz': '150903082.1412377808.303.70.utmcsr=duckduckgo.com|utmccn=(referral)|utmcmd=referral|utmcct=/l/',
 '_ga': 'GA1.2.282625269.1351702627',
 'optimizelyBuckets': '%7B%221970310409%22%3A%221962950399%22%7D',
 'optimizelyEndUserId': 'oeu1336076796179r0.8525460852353901',
 'optimizelySegments': '%7B%22197878113%22%3A%22none%22%2C%22197755715%22%3A%22search%22%2C%22197870141%22%3A%22ff%22%2C%22197869430%22%3A%22false%22%2C%22237061344%22%3A%22none%22%2C%22237321400%22%3A%22ff%22%2C%22237335298%22%3A%22referral%22%2C%22237485170%22%3A%22false%22%2C%22207211776%22%3A%22campaign%22%2C%22207213264%22%3A%22false%22%2C%22207203514%22%3A%22eoyfr2013%22%2C%22207203515%22%3A%22ff%22%2C%22245617832%22%3A%22none%22%2C%22245875585%22%3A%22referral%22%2C%22245677587%22%3A%22ff%22%2C%22246048108%22%3A%22false%22%2C%22743670347%22%3A%22true%22%2C%22704844240%22%3A%22true%22%2C%22869421433%22%3A%22true%22%2C%22697533825%22%3Atrue%2C%22246002457%22%3A%22direct%22%2C%22246073290%22%3A%22ff%22%2C%22245984388%22%3A%22false%22%2C%22246073289%22%3A%22none%22%2C%221709580498%22%3A%22ff%22%2C%221710521323%22%3A%22direct%22%2C%221706941468%22%3A%22false%22%2C%221706010671%22%3A%22none%22%2C%222020960057%22%3A%22false%22%2C%222007400026%22%3A%22ff%22%2C%222023620029%22%
 3A%22direct%22%2C%222015740038%22%3A%22none%22%2C%221867940538%22%3A%22true%22%7D'},
META:{'DOCUMENT_ROOT': '/data/www/moztrap.mozilla.org/moz-trap',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate',
 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.5',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': 'WT_FPC=id=63.245.220.240-2503254000.30218500:lv=1392795623402:ss=1392794403118; optimizelyEndUserId=oeu1336076796179r0.8525460852353901; optimizelyBuckets=%7B%221970310409%22%3A%221962950399%22%7D; __utma=150903082.282625269.1351702627.1412370493.1412377808.303; __utmz=150903082.1412377808.303.70.utmcsr=duckduckgo.com|utmccn=(referral)|utmcmd=referral|utmcct=/l/; optimizelySegments=%7B%22197878113%22%3A%22none%22%2C%22197755715%22%3A%22search%22%2C%22197870141%22%3A%22ff%22%2C%22197869430%22%3A%22false%22%2C%22237061344%22%3A%22none%22%2C%22237321400%22%3A%22ff%22%2C%22237335298%22%3A%22referral%22%2C%22237485170%22%3A%22false%22%2C%22207211776%22%3A%22campaign%22%2C%22207213264%22%3A%22false%22%2C%22207203514%22%3A%22eoyfr2013%22%2C%22207203515%22%3A%22ff%22%2C%22245617832%22%3A%22none%22%2C%22245875585%22%3A%22referral%22%2C%22245677587%22%3A%22ff%22%2C%22246048108%22%3A%22false%22%2C%22743670347%22%3A%22true%22%2C%22704844240%22%3A%22true%22%2C%22869421433
 %22%3A%22true%22%2C%22697533825%22%3Atrue%2C%22246002457%22%3A%22direct%22%2C%22246073290%22%3A%22ff%22%2C%22245984388%22%3A%22false%22%2C%22246073289%22%3A%22none%22%2C%221709580498%22%3A%22ff%22%2C%221710521323%22%3A%22direct%22%2C%221706941468%22%3A%22false%22%2C%221706010671%22%3A%22none%22%2C%222020960057%22%3A%22false%22%2C%222007400026%22%3A%22ff%22%2C%222023620029%22%3A%22direct%22%2C%222015740038%22%3A%22none%22%2C%221867940538%22%3A%22true%22%7D; __utmx=150903082.QhvDOg2zQqa9threH1A1kQ$71153379-29:1.SNKFJ4VqRziIW1TLqjhpAw$71153379-28:; __utmxx=150903082.QhvDOg2zQqa9threH1A1kQ$71153379-29:1383864788:15552000.SNKFJ4VqRziIW1TLqjhpAw$71153379-28:1398901249:15552000; _ga=GA1.2.282625269.1351702627',
 'HTTP_HOST': 'moztrap.mozilla.org',
 'HTTP_REFERER': 'https://moztrap.mozilla.org/manage/cases/?',
 'HTTP_SSLCLIENTCERTSTATUS': 'NoClientCert',
 'HTTP_SSLCLIENTCIPHER': 'SSL_DHE_RSA_WITH_AES_128_CBC_SHA, version=TLSv1.1, bits=128',
 'HTTP_SSLSESSIONID': '4E6E6FE8B6E538EF9647F07D07AA2CB120A6728F5EF63AC6D908225FD3B8BD86',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:34.0) Gecko/20100101 Firefox/34.0',
 'HTTP_X_CLUSTER_CLIENT_IP': '70.68.38.128',
 'HTTP_X_FORWARDED_FOR': '70.68.38.128',
 'HTTP_X_SSL': 'On',
 'PATH_INFO': u'/users/login/',
 'PATH_TRANSLATED': '/data/www/moztrap.mozilla.org/moz-trap/moztrap/deploy/vendor_wsgi.py/users/login/',
 'QUERY_STRING': 'next=/manage/cases/',
 'REMOTE_ADDR': '10.22.81.208',
 'REMOTE_PORT': '45737',
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/users/login/?next=/manage/cases/',
 'SCRIPT_FILENAME': '/data/www/moztrap.mozilla.org/moz-trap/moztrap/deploy/vendor_wsgi.py',
 'SCRIPT_NAME': u'',
 'SCRIPT_URI': 'http://moztrap.mozilla.org/users/login/',
 'SCRIPT_URL': '/users/login/',
 'SERVER_ADDR': '10.22.81.52',
 'SERVER_ADMIN': '[e-mail]',
 'SERVER_NAME': 'moztrap.mozilla.org',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '',
 'SERVER_SOFTWARE': 'Apache',
 'apache.version': (2, 2, 15),
 'mod_wsgi.application_group': 'moztrap.mozilla.org|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.daemon_connects': '1',
 'mod_wsgi.daemon_restarts': '0',
 'mod_wsgi.daemon_start': '1412637518123376',
 'mod_wsgi.enable_sendfile': '0',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '81',
 'mod_wsgi.process_group': 'moztrap',
 'mod_wsgi.queue_start': '1412637518123231',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.request_start': '1412637518122889',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.script_start': '1412637518123407',
 'mod_wsgi.version': (4, 2, 7),
 'newrelic.app_name': 'moztrap.mozilla.org',
 'wsgi.errors': <mod_wsgi.Log object at 0x7f537a6bcb28>,
 'wsgi.file_wrapper': <type 'mod_wsgi.FileWrapper'>,
 'wsgi.input': <newrelic.api.web_transaction.WSGIInputWrapper object at 0x7f537af2c6d0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

Comment 7

[Matt Pressman [:mpressman]]

moztrap1.db.phx1.mozilla.com is the master and it is running in r/w mode
mysql> show variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | OFF   |
+---------------+-------+

Comment 8

[Chris Turra [:cturra]]

as discussed on irc, this appears to only be happening in scl3. in that datacenter, the database-rw server is reporting read_only ON.

 mysql> show variables like 'read_only'\G
 *************************** 1. row ***************************
 Variable_name: read_only
         Value: ON
 1 row in set (0.00 sec)

Comment 9

[Chris Turra [:cturra]]

time for an update. we've figured out what's going on here. all the traffic for moztrap is being served out of scl3 because of the phx1 network outage earlier today. this site is designed to be a manual failover, which meant the database for it needed to be manually flipped out of read_only mode. :mpressman has now done that for us, so moztrap should all be functioning as expected once again.

there is some further work we'll want to do tomorrow to fail this back to phx1. specifically, the databases will need to be re-synced so we don't run into a split brain situation. i will co-ordinate that with the dba's outside of this bug. 

for now, i am marking this bug as r/fixed.

Comment 10

[Tony Chung [:tchung]]

Woohoo Thanks Chris!   Marcia, Peter, can you confirm we are good?

Comment 11

[Marcia Knous [:marcia]]

Editing seems to work for me now without the error.

Chris: Will the fallback and resync be transparent or will the tool be down? If there will be some downtime would be good to know this ahead of time if possible. Thanks.

Comment 12

[Chris Turra [:cturra]]

(In reply to Marcia Knous [:marcia - use needinfo] from comment #11)
> 
> Chris: Will the fallback and resync be transparent or will the tool be down?
> If there will be some downtime would be good to know this ahead of time if
> possible. Thanks.

failback will be transparent.

Comment 13

[Peter Bylenga [:PBylenga]]

We were able to pass/fail our cases now as well.  Also I can logout/login with Persona without issue now on moztrap.

Thanks!