731310 - (talos-r3-fed-027) talos-r3-fed-027 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[[e-mail] (:bhearsum)]

offline again

Comment 2

[[e-mail] (:bhearsum)]

Back in production.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

offline again.

Comment 4

[Jake Watkins [:dividehex]]

grey screen | rebooted

Comment 5

[Mike Taylor [:bear]]

returned to production

Comment 6

[Armen [:armenzg]]

[54] talos-r3-fed-027.build.scl1:PING is CRITICAL: PING CRITICAL - Packet loss = 100%

Comment 7

[Armen [:armenzg]]

It came back by itself 35 minutes later.

nagios-sjc1: talos-r3-fed-027.build.scl1:PING is OK: PING OK - Packet loss = 0%, RTA = 0.71 ms

Comment 8

[[e-mail] (:bhearsum)]

Needs a reboot.

Comment 9

[Vinh Hua [:vinh]]

Rebooted.

Comment 10

[Kim Moir [:kmoir] ET]

Needs a reboot.

Comment 11

[[e-mail] (:bhearsum)]

needs a reboot

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Needs a reboot.

Comment 13

[Armen [:armenzg]]

Taking jobs.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Needs a reboot.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Back in action.

Comment 16

[Hal Wine [:hwine] use NI!]

down since 12-27-2012 13:46:58

Comment 17

[John Hopkins (:jhopkins)]

time since last job: 1 day, 4:34:19
not responding to pings

Comment 18

[John Hopkins (:jhopkins)]

Back online and performing builds.

Comment 19

[[e-mail] (:bhearsum)]

Back in production.

Comment 20

[John Hopkins (:jhopkins)]

recent jobs are green

Comment 21

[[e-mail] (:bhearsum)]

This machine is having issues when connecting to its master:
2014-02-19 09:05:34-0800 [-] Log opened.
2014-02-19 09:05:34-0800 [-] twistd 10.2.0 (/tools/buildbot-0.8.4-pre-moz2/bin/python 2.6.2) starting up.
2014-02-19 09:05:34-0800 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2014-02-19 09:05:34-0800 [-] Starting factory <buildslave.bot.BotFactory instance at 0xb71779cc>
2014-02-19 09:05:34-0800 [-] Connecting to buildbot-master67.srv.releng.use1.mozilla.com:9201
2014-02-19 09:05:34-0800 [-] Watching /home/cltbld/talos-slave/shutdown.stamp's mtime to initiate shutdown
2014-02-19 09:07:28-0800 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2014-02-19 09:07:28-0800 [Broker,client] While trying to connect:
	Traceback from remote host -- Traceback (most recent call last):
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/spread/pb.py", line 1346, in remote_respond
	    d = self.portal.login(self, mind, IPerspective)
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/cred/portal.py", line 116, in login
	    ).addCallback(self.realm.requestAvatar, mind, *interfaces
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/internet/defer.py", line 260, in addCallback
	    callbackKeywords=kw)
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
	    self._runCallbacks()
	--- <exception caught here> ---
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/buildbot-0.8.2_hg_f23f5672becd_production_0.8-py2.7.egg/buildbot/master.py", line 498, in requestAvatar
	    p = self.botmaster.getPerspective(mind, avatarID)
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/buildbot-0.8.2_hg_f23f5672becd_production_0.8-py2.7.egg/buildbot/master.py", line 364, in getPerspective
	    d = sl.slave.callRemote("print", "master got a duplicate connection; keeping this one")
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/spread/pb.py", line 328, in callRemote
	    _name, args, kw)
	  File "/builds/buildbot/tests1-linux/lib/python2.7/site-packages/twisted/spread/pb.py", line 807, in _sendMessage
	    raise DeadReferenceError("Calling Stale Broker")
	twisted.spread.pb.DeadReferenceError: Calling Stale Broker
	
2014-02-19 09:07:28-0800 [Broker,client] Lost connection to buildbot-master67.srv.releng.use1.mozilla.com:9201
2014-02-19 09:07:28-0800 [Broker,client] Stopping factory <buildslave.bot.BotFactory instance at 0xb71779cc>
2014-02-19 09:07:28-0800 [-] Main loop terminated.
2014-02-19 09:07:28-0800 [-] Server Shut Down.

Comment 22

[[e-mail] (:bhearsum)]

Found and fixed the problem over in bug 974493.