754007 - mdn: OperationalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Comment 1

[Luke Crouch [:groovecoder]]

Prod db is running in read-only mode?!

Comment 2

[raymond [:retornam] (inactive)]

Traceback (most recent call last):

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)

  File "/data/www/django/developer.mozilla.org/kuma/apps/demos/views.py", line 222, in launch
    submission.launches.increment(request)

  File "/data/www/django/developer.mozilla.org/kuma/apps/actioncounters/fields.py", line 93, in increment
    counter = self._get_counter_for_request(request)

  File "/data/www/django/developer.mozilla.org/kuma/apps/actioncounters/fields.py", line 85, in _get_counter_for_request
    self.instance, self.name, request, do_create)

  File "/data/www/django/developer.mozilla.org/kuma/apps/actioncounters/models.py", line 52, in get_unique_for_request
    total=0))

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/manager.py", line 135, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)

  File "/data/www/django/developer.mozilla.org/kuma/apps/actioncounters/models.py", line 101, in save
    super(ActionCounterUnique, self).save(*args, **kwargs)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/base.py", line 458, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/base.py", line 551, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/query.py", line 1524, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/sql/compiler.py", line 788, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/models/sql/compiler.py", line 732, in execute_sql
    cursor.execute(sql, params)

  File "/data/www/django/developer.mozilla.org/kuma/vendor/src/django/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)

  File "/usr/lib/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)

  File "/usr/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue

OperationalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')

Comment 3

[Jake Maul [:jakem]]

I can't replicate this error on the live site... was this a one-time thing or ongoing?

Comment 4

[Sheeri Cabral [:sheeri]]

Fixed for now - the problem was that for some reason, the load balancer failed writes over to the slave. Which was set to read_only=OFF. I've changed that on the slave, but we need to come up with a good policy on this.

Comment 5

[Sheeri Cabral [:sheeri]]

[16:45:05] <@cshields> sheeri: the only way I've found to clear a passive monitor failure in a single-node pool is by disabling and re-enabling that node.  The only way to do that is to add another (fake) node to the pool and then disable, then enable and remov that host
[16:45:09] <@cshields> which is now done
[16:45:16] <@cshields> and passive monitoring should always be turned off
[16:45:50] <sheeri> cshields and in fact, the rw vip now goes to developer1

All back to normal now!

Comment 6

[raymond [:retornam] (inactive)]

marking as fixed since I haven't gotten  traceback emails since  yesterday