Closed
Bug 754813
Opened 13 years ago
Closed 13 years ago
Build scheduler MySQL errors during sendchange ('The MySQL server is running with the --read-only option so it cannot execute this statement')
Categories
(Data & BI Services Team :: DB: MySQL, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: emorley, Assigned: scabral)
References
Details
On multiple trees:
Linux x86-64 mozilla-central nightly on 2012-05-14 03:05:23 PDT for push ac968ff4fe41
slave: linux64-ix-slave10
{
========= Started sendchange failed (results: 2, elapsed: 1 mins, 16 secs) (at 2012-05-14 04:02:00.183870) =========
master: buildbot-master36.build.mozilla.org:9301
branch: mozilla-central-linux64-pgo-talos
revision: ac968ff4fe41
comments:
user: sendchange
files: ['http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
properties: [('buildid', '20120514030523'), ('pgo_build', True), ('builduid', u'e94c75f7f5aa40c285cf96d1fc897df9'), ('nightly_build', True)]
python /builds/slave/m-cen-lnx64-ntly/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 5 --stdout-regexp 'change sent successfully' buildbot sendchange --master buildbot-master36.build.mozilla.org:9301 --username sendchange --branch mozilla-central-linux64-pgo-talos --revision ac968ff4fe41 --property buildid:20120514030523 --property pgo_build:True --property builduid:e94c75f7f5aa40c285cf96d1fc897df9 --property nightly_build:True http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2
in dir /builds/slave/m-cen-lnx64-ntly/build (timeout 1200 secs)
watching logfiles {}
argv: ['python', '/builds/slave/m-cen-lnx64-ntly/tools/buildfarm/utils/retry.py', '-s', '5', '-t', '1800', '-r', '5', '--stdout-regexp', 'change sent successfully', 'buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', u'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
environment:
CVS_RSH=ssh
G_BROKEN_FILENAMES=1
HISTSIZE=1000
HOME=/home/cltbld
HOSTNAME=linux64-ix-slave10.build.scl1.mozilla.com
INPUTRC=/etc/inputrc
LANG=en_US.UTF-8
LESSOPEN=|/usr/bin/lesspipe.sh %s
LOGNAME=cltbld
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
MAIL=/var/spool/mail/cltbld
PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
PWD=/builds/slave/m-cen-lnx64-ntly/build
SHELL=/bin/bash
SHLVL=1
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
TERM=linux
USER=cltbld
_=/tools/python/bin/python
using PTY: False
retry: Calling <function run_with_timeout at 0x2aaaaed1bc80> with args: (['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'], 1800, 'change sent successfully', None, False, True), kwargs: {}, attempt #1
Executing: ['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
retry: Failed, sleeping 5 seconds before retrying
retry: Calling <function run_with_timeout at 0x2aaaaed1bc80> with args: (['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'], 1800, 'change sent successfully', None, False, True), kwargs: {}, attempt #2
Executing: ['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
Process stdio:
NOTE: --username/-u is deprecated: use --who/-W'
change(s) NOT sent, something went wrong: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/banana.py", line 153, in gotItem
self.callExpressionReceived(item)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/banana.py", line 116, in callExpressionReceived
self.expressionReceived(obj)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 514, in expressionReceived
method(*sexp[1:])
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 826, in proto_message
self._recvMessage(self.localObjectForID, requestID, objectID, message, answerRequired, netArgs, netKw)
--- <exception caught here> ---
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 840, in _recvMessage
netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 225, in perspectiveMessageReceived
state = method(*args, **kw)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/changes/pb.py", line 43, in perspective_addChange
self.changemaster.addChange(change)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/changes/manager.py", line 114, in addChange
self.parent.addChange(change)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/master.py", line 1150, in addChange
self.db.addChangeToDatabase(change)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 308, in addChangeToDatabase
self.runInteractionNow(self._txn_addChangeToDatabase, change)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 212, in runInteractionNow
return self._runInteractionNow(interaction, *args, **kwargs)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 237, in _runInteractionNow
result = interaction(c, *args, **kwargs)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 326, in _txn_addChangeToDatabase
t.execute(q, values)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')
]
...
...
}
Example full log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724077&tree=Firefox
Seems like the build scheduler should handle this more gracefully too, and/or output something more useful for TBPL, rather than just turning orange with nothing in the "FAIL:..." box.
Reporter | ||
Comment 1•13 years ago
|
||
Reporter | ||
Comment 2•13 years ago
|
||
Reporter | ||
Comment 3•13 years ago
|
||
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723983&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724120&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724113&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724114&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724164&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724196&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724145&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723973&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723972&tree=Mozilla-Inbound
Reporter | ||
Comment 4•13 years ago
|
||
New builds aren't being started either.
eg: the current inbound tip:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=6678097e44e5
buildapi says revision not found etc (https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/6678097e44e5)
Reporter | ||
Comment 5•13 years ago
|
||
Have closed mozilla-central, inbound, fx-team, aurora, beta, esr and try at 1305 UTC+1, since builds aren't getting started.
Comment 6•13 years ago
|
||
this is due to the master db in buildbot failing, we're looking into why the failover hit a read-only setting..
Updated•13 years ago
|
Assignee: server-ops-releng → server-ops-database
Component: Server Operations: RelEng → Server Operations: Database
QA Contact: arich → cshields
Assignee | ||
Updated•13 years ago
|
Assignee: server-ops-database → scabral
Assignee | ||
Comment 7•13 years ago
|
||
I have set read_only=OFF on buildbot2.
I have also updated puppet to make read_only=off on machines that are master/master, and set builbot to be master/master in puppet.
https://bugzilla.mozilla.org/show_bug.cgi?id=754820 is what we need to do so we don't get bitten by this again.
Leaving this open because we have to fail back to buildbot1....
See Also: → 754820
Reporter | ||
Comment 8•13 years ago
|
||
Looking good; the builds I retriggered are succeeding.
Thanks Sheeri, Amy, Corey & everyone else :-D
Reporter | ||
Comment 9•13 years ago
|
||
Oh forgot the important part: the trees have been reopened.
Assignee | ||
Comment 10•13 years ago
|
||
anything left to do on this particular bug? or is it resolved/fixed?
Reporter | ||
Comment 11•13 years ago
|
||
It's fixed as far as I'm concerned :-)
(I had only left open for the last sentence of comment 7, but I presume I'd misinterpreted it as a needed followup, given comment 10).
Comment 12•13 years ago
|
||
all clear from the buildduty point of view
Assignee | ||
Updated•13 years ago
|
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Updated•10 years ago
|
Product: mozilla.org → Data & BI Services Team
You need to log in
before you can comment on or make changes to this bug.
Description
•