Closed Bug 754813 Opened 13 years ago Closed 13 years ago

Build scheduler MySQL errors during sendchange ('The MySQL server is running with the --read-only option so it cannot execute this statement')

Categories

(Data & BI Services Team :: DB: MySQL, task)

All
Other
task
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: scabral)

References

Details

On multiple trees: Linux x86-64 mozilla-central nightly on 2012-05-14 03:05:23 PDT for push ac968ff4fe41 slave: linux64-ix-slave10 { ========= Started sendchange failed (results: 2, elapsed: 1 mins, 16 secs) (at 2012-05-14 04:02:00.183870) ========= master: buildbot-master36.build.mozilla.org:9301 branch: mozilla-central-linux64-pgo-talos revision: ac968ff4fe41 comments: user: sendchange files: ['http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'] properties: [('buildid', '20120514030523'), ('pgo_build', True), ('builduid', u'e94c75f7f5aa40c285cf96d1fc897df9'), ('nightly_build', True)] python /builds/slave/m-cen-lnx64-ntly/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 5 --stdout-regexp 'change sent successfully' buildbot sendchange --master buildbot-master36.build.mozilla.org:9301 --username sendchange --branch mozilla-central-linux64-pgo-talos --revision ac968ff4fe41 --property buildid:20120514030523 --property pgo_build:True --property builduid:e94c75f7f5aa40c285cf96d1fc897df9 --property nightly_build:True http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2 in dir /builds/slave/m-cen-lnx64-ntly/build (timeout 1200 secs) watching logfiles {} argv: ['python', '/builds/slave/m-cen-lnx64-ntly/tools/buildfarm/utils/retry.py', '-s', '5', '-t', '1800', '-r', '5', '--stdout-regexp', 'change sent successfully', 'buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', u'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'] environment: CVS_RSH=ssh G_BROKEN_FILENAMES=1 HISTSIZE=1000 HOME=/home/cltbld HOSTNAME=linux64-ix-slave10.build.scl1.mozilla.com INPUTRC=/etc/inputrc LANG=en_US.UTF-8 LESSOPEN=|/usr/bin/lesspipe.sh %s LOGNAME=cltbld LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: MAIL=/var/spool/mail/cltbld PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin PWD=/builds/slave/m-cen-lnx64-ntly/build SHELL=/bin/bash SHLVL=1 SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass TERM=linux USER=cltbld _=/tools/python/bin/python using PTY: False retry: Calling <function run_with_timeout at 0x2aaaaed1bc80> with args: (['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'], 1800, 'change sent successfully', None, False, True), kwargs: {}, attempt #1 Executing: ['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'] retry: Failed, sleeping 5 seconds before retrying retry: Calling <function run_with_timeout at 0x2aaaaed1bc80> with args: (['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'], 1800, 'change sent successfully', None, False, True), kwargs: {}, attempt #2 Executing: ['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'] Process stdio: NOTE: --username/-u is deprecated: use --who/-W' change(s) NOT sent, something went wrong: [Failure instance: Traceback from remote host -- Traceback (most recent call last): File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/banana.py", line 153, in gotItem self.callExpressionReceived(item) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/banana.py", line 116, in callExpressionReceived self.expressionReceived(obj) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 514, in expressionReceived method(*sexp[1:]) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 826, in proto_message self._recvMessage(self.localObjectForID, requestID, objectID, message, answerRequired, netArgs, netKw) --- <exception caught here> --- File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 840, in _recvMessage netResult = object.remoteMessageReceived(self, message, netArgs, netKw) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 225, in perspectiveMessageReceived state = method(*args, **kw) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/changes/pb.py", line 43, in perspective_addChange self.changemaster.addChange(change) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/changes/manager.py", line 114, in addChange self.parent.addChange(change) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/master.py", line 1150, in addChange self.db.addChangeToDatabase(change) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 308, in addChangeToDatabase self.runInteractionNow(self._txn_addChangeToDatabase, change) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 212, in runInteractionNow return self._runInteractionNow(interaction, *args, **kwargs) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 237, in _runInteractionNow result = interaction(c, *args, **kwargs) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 326, in _txn_addChangeToDatabase t.execute(q, values) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute self.errorhandler(self, exc, value) File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.OperationalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement') ] ... ... } Example full log: https://tbpl.mozilla.org/php/getParsedLog.php?id=11724077&tree=Firefox Seems like the build scheduler should handle this more gracefully too, and/or output something more useful for TBPL, rather than just turning orange with nothing in the "FAIL:..." box.
New builds aren't being started either. eg: the current inbound tip: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=6678097e44e5 buildapi says revision not found etc (https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/6678097e44e5)
Have closed mozilla-central, inbound, fx-team, aurora, beta, esr and try at 1305 UTC+1, since builds aren't getting started.
this is due to the master db in buildbot failing, we're looking into why the failover hit a read-only setting..
Assignee: server-ops-releng → server-ops-database
Component: Server Operations: RelEng → Server Operations: Database
QA Contact: arich → cshields
Assignee: server-ops-database → scabral
I have set read_only=OFF on buildbot2. I have also updated puppet to make read_only=off on machines that are master/master, and set builbot to be master/master in puppet. https://bugzilla.mozilla.org/show_bug.cgi?id=754820 is what we need to do so we don't get bitten by this again. Leaving this open because we have to fail back to buildbot1....
See Also: → 754820
Looking good; the builds I retriggered are succeeding. Thanks Sheeri, Amy, Corey & everyone else :-D
Oh forgot the important part: the trees have been reopened.
anything left to do on this particular bug? or is it resolved/fixed?
It's fixed as far as I'm concerned :-) (I had only left open for the last sentence of comment 7, but I presume I'd misinterpreted it as a needed followup, given comment 10).
all clear from the buildduty point of view
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.