793720 - balrog dev environment is getting a lot of "The MySQL server is running with the --read-only option" errors

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[[e-mail] (:bhearsum)]

I noticed a bunch of failures trying to submit data to Balrog this morning. After poking around the server logs I found a bunch of:
528332-TransactionError: ("(OperationalError) (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')",)


in the logs. The first one is at 05:43:53 PT this morning.

I'm also seeing some errors prior to that about data_version not matching the previous one -- those are coming from the app when server data changes mid-update, so could potentially indicate some sort of replication issue? that's pure speculation on my part, though.

Comment 1

[Sheeri Cabral [:sheeri]]

copying from IRC - we did have dev issues this morning...


[11:09:59] <bhearsum> mysql://aus4_dev:[e-mail]/aus4_dev
[11:10:44] <sheeri> interesting, 'cause I see connections from aus4_dev@ 10.8.70.208 on the master server, which is set to read_only=OFF
[11:10:53] <sheeri> the slave has read_only=ON on it
[11:10:58] <sheeri> let me double-check the load balancers
[11:11:13] <sheeri> bhearsum also is this as of now, or was it before about an hour ago, say, from Friday onwards?
[11:11:17] <bhearsum> the connections should be coming from 10.8.33.19 and 20
[11:11:23] <bhearsum> one sec, i'll find the first one
[11:11:56] <bhearsum> afaict the first one is at 05:43:53 PT this morning
[11:14:08] <bhearsum> i'm also seeing some errors prior to that about data_version not matching the previous one -- those are coming from the app when server data changes mid-update, so could potentially indicate some sort of replication issue? that's pure speculation on my part, though
[11:14:59] <bhearsum> filed https://bugzilla.mozilla.org/show_bug.cgi?id=793720
[11:15:01] <firebot> Bug 793720 nor, --, ---, server-ops-database, NEW, balrog dev environment is getting a lot of "The MySQL server is running with the --read-only option"
[11:15:45] <sheeri> as for the timing, are they still happening? we were having dev issues.
[11:15:59] <bhearsum> i think so, one sec
[11:16:06] <sheeri> we had to re-sync, so that's probably what the data_version errors were?
[11:16:19] <bhearsum> ahhh, probably
[11:16:32] <bhearsum> the most recent failure i see is @ 06:29:25 PT
[11:16:47] <bhearsum> but these only happen during nightly builds...
[11:16:54] <bhearsum> let me try doing a manual request to see what happens
[11:17:34] <sheeri> yes, please.

Comment 2

[Sheeri Cabral [:sheeri]]

[11:20:11] <bhearsum> okay, i just resubmitted one of the failed ones without issue
[11:23:01] armenzg_brb is now known as armenzg
[11:24:49] <sheeri> bhearsum OK. want to leave it open until tomorrow and make sure none of the nighties fail again?
[11:26:02] <bhearsum> ehh, that's ok
[11:26:09] <bhearsum> i can re-open if it has issues
[11:26:22] <bhearsum> so this was just a transient failure on the mysql backend?
[11:26:29] <sheeri> yes
[11:27:34] <bhearsum> ok
[11:27:38] <bhearsum> thanks for looking into it so quickly!
[11:27:49] <sheeri> np