I noticed a bunch of failures trying to submit data to Balrog this morning. After poking around the server logs I found a bunch of: 528332-TransactionError: ("(OperationalError) (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')",) in the logs. The first one is at 05:43:53 PT this morning. I'm also seeing some errors prior to that about data_version not matching the previous one -- those are coming from the app when server data changes mid-update, so could potentially indicate some sort of replication issue? that's pure speculation on my part, though.
copying from IRC - we did have dev issues this morning... [11:09:59] <bhearsum> mysql://aus4_dev:email@example.com/aus4_dev [11:10:44] <sheeri> interesting, 'cause I see connections from aus4_dev@ 10.8.70.208 on the master server, which is set to read_only=OFF [11:10:53] <sheeri> the slave has read_only=ON on it [11:10:58] <sheeri> let me double-check the load balancers [11:11:13] <sheeri> bhearsum also is this as of now, or was it before about an hour ago, say, from Friday onwards? [11:11:17] <bhearsum> the connections should be coming from 10.8.33.19 and 20 [11:11:23] <bhearsum> one sec, i'll find the first one [11:11:56] <bhearsum> afaict the first one is at 05:43:53 PT this morning [11:14:08] <bhearsum> i'm also seeing some errors prior to that about data_version not matching the previous one -- those are coming from the app when server data changes mid-update, so could potentially indicate some sort of replication issue? that's pure speculation on my part, though [11:14:59] <bhearsum> filed https://bugzilla.mozilla.org/show_bug.cgi?id=793720 [11:15:01] <firebot> Bug 793720 nor, --, ---, server-ops-database, NEW, balrog dev environment is getting a lot of "The MySQL server is running with the --read-only option" [11:15:45] <sheeri> as for the timing, are they still happening? we were having dev issues. [11:15:59] <bhearsum> i think so, one sec [11:16:06] <sheeri> we had to re-sync, so that's probably what the data_version errors were? [11:16:19] <bhearsum> ahhh, probably [11:16:32] <bhearsum> the most recent failure i see is @ 06:29:25 PT [11:16:47] <bhearsum> but these only happen during nightly builds... [11:16:54] <bhearsum> let me try doing a manual request to see what happens [11:17:34] <sheeri> yes, please.
[11:20:11] <bhearsum> okay, i just resubmitted one of the failed ones without issue [11:23:01] armenzg_brb is now known as armenzg [11:24:49] <sheeri> bhearsum OK. want to leave it open until tomorrow and make sure none of the nighties fail again? [11:26:02] <bhearsum> ehh, that's ok [11:26:09] <bhearsum> i can re-open if it has issues [11:26:22] <bhearsum> so this was just a transient failure on the mysql backend? [11:26:29] <sheeri> yes [11:27:34] <bhearsum> ok [11:27:38] <bhearsum> thanks for looking into it so quickly! [11:27:49] <sheeri> np