666050 - test-master01 stale broker exceptions

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Aki Sasaki (not active)]

This has been happening for a while now.

Exception in /builds/buildbot/tests-master/twistd.log:
2011-06-21 14:59:14-0700 [Broker,6384,10.250.48.202] Unhandled Error
	Traceback (most recent call last):
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/spread/pb.py", line 1346, in remote_respond
	    d = self.portal.login(self, mind, IPerspective)
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/cred/portal.py", line 116, in login
	    ).addCallback(self.realm.requestAvatar, mind, *interfaces
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/defer.py", line 260, in addCallback
	    callbackKeywords=kw)
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/master.py", line 474, in requestAvatar
	    p = self.botmaster.getPerspective(mind, avatarID)
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/master.py", line 344, in getPerspective
	    d = sl.slave.callRemote("print", "master got a duplicate connection; keeping this one")
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/spread/pb.py", line 328, in callRemote
	    _name, args, kw)
	  File "/tools/buildbot-0.8.2/lib/python2.6/site-packages/twisted/spread/pb.py", line 807, in _sendMessage
	    raise DeadReferenceError("Calling Stale Broker")
	twisted.spread.pb.DeadReferenceError: Calling Stale Broker

Comment 1

[Aki Sasaki (not active)]

I think test-master01 reconfigs have required graceful master restarts for a very similar time frame.

Comment 2

[Mike Taylor [:bear]]

going to create a new test-master to see if a buildbot upgrade removes this as an issue.  bug 666737 filed to track the new vm for the test master - after it's ready we will migrate all slaves to it and retire this one.

Comment 3

[Aki Sasaki (not active)]

I haven't seen any since last week.