Build scheduler MySQL errors during sendchange ('The MySQL server is running with the --read-only option so it cannot execute this statement')

RESOLVED FIXED

Status

Data & BI Services Team
DB: MySQL
--
blocker
RESOLVED FIXED
6 years ago
3 years ago

People

(Reporter: emorley, Assigned: sheeri)

Tracking

Details

(Reporter)

Description

6 years ago
On multiple trees:

Linux x86-64 mozilla-central nightly on 2012-05-14 03:05:23 PDT for push ac968ff4fe41

slave: linux64-ix-slave10

{
========= Started sendchange failed (results: 2, elapsed: 1 mins, 16 secs) (at 2012-05-14 04:02:00.183870) =========
    master: buildbot-master36.build.mozilla.org:9301
    branch: mozilla-central-linux64-pgo-talos
    revision: ac968ff4fe41
    comments: 
    user: sendchange
    files: ['http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
    properties: [('buildid', '20120514030523'), ('pgo_build', True), ('builduid', u'e94c75f7f5aa40c285cf96d1fc897df9'), ('nightly_build', True)]
python /builds/slave/m-cen-lnx64-ntly/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 5 --stdout-regexp 'change sent successfully' buildbot sendchange --master buildbot-master36.build.mozilla.org:9301 --username sendchange --branch mozilla-central-linux64-pgo-talos --revision ac968ff4fe41 --property buildid:20120514030523 --property pgo_build:True --property builduid:e94c75f7f5aa40c285cf96d1fc897df9 --property nightly_build:True http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2
 in dir /builds/slave/m-cen-lnx64-ntly/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', '/builds/slave/m-cen-lnx64-ntly/tools/buildfarm/utils/retry.py', '-s', '5', '-t', '1800', '-r', '5', '--stdout-regexp', 'change sent successfully', 'buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', u'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
 environment:
  CVS_RSH=ssh
  G_BROKEN_FILENAMES=1
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=linux64-ix-slave10.build.scl1.mozilla.com
  INPUTRC=/etc/inputrc
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
  MAIL=/var/spool/mail/cltbld
  PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
  PWD=/builds/slave/m-cen-lnx64-ntly/build
  SHELL=/bin/bash
  SHLVL=1
  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
  TERM=linux
  USER=cltbld
  _=/tools/python/bin/python
 using PTY: False
retry: Calling <function run_with_timeout at 0x2aaaaed1bc80> with args: (['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'], 1800, 'change sent successfully', None, False, True), kwargs: {}, attempt #1
Executing: ['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
retry: Failed, sleeping 5 seconds before retrying
retry: Calling <function run_with_timeout at 0x2aaaaed1bc80> with args: (['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2'], 1800, 'change sent successfully', None, False, True), kwargs: {}, attempt #2
Executing: ['buildbot', 'sendchange', '--master', 'buildbot-master36.build.mozilla.org:9301', '--username', 'sendchange', '--branch', 'mozilla-central-linux64-pgo-talos', '--revision', 'ac968ff4fe41', '--property', 'buildid:20120514030523', '--property', 'pgo_build:True', '--property', 'builduid:e94c75f7f5aa40c285cf96d1fc897df9', '--property', 'nightly_build:True', 'http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1336989923/firefox-15.0a1.en-US.linux-x86_64.tar.bz2']
Process stdio:
NOTE: --username/-u is deprecated: use --who/-W'
change(s) NOT sent, something went wrong: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/banana.py", line 153, in gotItem
    self.callExpressionReceived(item)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/banana.py", line 116, in callExpressionReceived
    self.expressionReceived(obj)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 514, in expressionReceived
    method(*sexp[1:])
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 826, in proto_message
    self._recvMessage(self.localObjectForID, requestID, objectID, message, answerRequired, netArgs, netKw)
--- <exception caught here> ---
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 840, in _recvMessage
    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/twisted/spread/pb.py", line 225, in perspectiveMessageReceived
    state = method(*args, **kw)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/changes/pb.py", line 43, in perspective_addChange
    self.changemaster.addChange(change)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/changes/manager.py", line 114, in addChange
    self.parent.addChange(change)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/master.py", line 1150, in addChange
    self.db.addChangeToDatabase(change)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 308, in addChangeToDatabase
    self.runInteractionNow(self._txn_addChangeToDatabase, change)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 212, in runInteractionNow
    return self._runInteractionNow(interaction, *args, **kwargs)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 237, in _runInteractionNow
    result = interaction(c, *args, **kwargs)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/buildbot-0.8.2_hg_a2045101fe7a_production_0.8-py2.6.egg/buildbot/db/connector.py", line 326, in _txn_addChangeToDatabase
    t.execute(q, values)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/builds/buildbot/build_scheduler/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')
]
...
...
}

Example full log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724077&tree=Firefox

Seems like the build scheduler should handle this more gracefully too, and/or output something more useful for TBPL, rather than just turning orange with nothing in the "FAIL:..." box.
(Reporter)

Comment 1

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724188&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724087&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724080&tree=Mozilla-Inbound
(Reporter)

Comment 2

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724086&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724077&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724181&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723978&tree=Firefox
(Reporter)

Comment 3

6 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723983&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724120&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724113&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724114&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724164&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724196&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11724145&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723973&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=11723972&tree=Mozilla-Inbound
(Reporter)

Comment 4

6 years ago
New builds aren't being started either.

eg: the current inbound tip:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=6678097e44e5

buildapi says revision not found etc (https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/6678097e44e5)
(Reporter)

Comment 5

6 years ago
Have closed mozilla-central, inbound, fx-team, aurora, beta, esr and try at 1305 UTC+1, since builds aren't getting started.
this is due to the master db in buildbot failing, we're looking into why the failover hit a read-only setting..
Assignee: server-ops-releng → server-ops-database
Component: Server Operations: RelEng → Server Operations: Database
QA Contact: arich → cshields
(Assignee)

Updated

6 years ago
Assignee: server-ops-database → scabral
(Assignee)

Comment 7

6 years ago
I have set read_only=OFF on buildbot2.

I have also updated puppet to make read_only=off on machines that are master/master, and set builbot to be master/master in puppet.

https://bugzilla.mozilla.org/show_bug.cgi?id=754820 is what we need to do so we don't get bitten by this again.

Leaving this open because we have to fail back to buildbot1....
See Also: → bug 754820
(Reporter)

Comment 8

6 years ago
Looking good; the builds I retriggered are succeeding.

Thanks Sheeri, Amy, Corey & everyone else :-D
(Reporter)

Comment 9

6 years ago
Oh forgot the important part: the trees have been reopened.
(Assignee)

Comment 10

6 years ago
anything left to do on this particular bug? or is it resolved/fixed?
(Reporter)

Comment 11

6 years ago
It's fixed as far as I'm concerned :-)

(I had only left open for the last sentence of comment 7, but I presume I'd misinterpreted it as a needed followup, given comment 10).

Comment 12

6 years ago
all clear from the buildduty point of view
(Assignee)

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.