Closed Bug 948044 Opened 11 years ago Closed 10 years ago

WEIRD: Builder.detached messages during reconfig

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: bhearsum, Unassigned)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2843] )

Got a ton of these which seems to be hanging the reconfig on bm54, all for the same slave (tst-linux64-ec2-368) 2013-12-09 10:56:04-0800 [Broker,77250,10.132.56.221] WEIRD: Builder.detached(<BuildSlave 'tst-linux64-ec2-368'>) (tst-linux64-ec2-368) not in attaching_slaves([]) or slaves([<SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-332'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-365'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-436'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-338'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-330'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-401'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-334'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-475'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-496'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-313'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-403'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-369'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-324'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-411'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-362'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-408'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-414'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-495'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-396'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-372'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-352'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-394'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-321'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-344'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-301'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-322'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-351'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-410'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-494'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-410'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-432'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-422'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-492'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-323'>]) The twistd.log on the slave looks normal, though I'm surprised the slave isn't running or even attempting to: 2013-12-09 10:56:02-0800 [Broker,client] lost remote 2013-12-09 10:56:02-0800 [Broker,client] lost remote 2013-12-09 10:56:02-0800 [Broker,client] lost remote 2013-12-09 10:56:02-0800 [Broker,client] lost remote 2013-12-09 10:56:02-0800 [Broker,client] lost remote 2013-12-09 10:56:02-0800 [Broker,client] lost remote 2013-12-09 10:56:02-0800 [Broker,client] Lost connection to buildbot-master54.srv.releng.usw2.mozilla.com:9201 2013-12-09 10:56:02-0800 [Broker,client] Stopping factory <buildslave.bot.BotFactory instance at 0x7f90f391eb00> 2013-12-09 10:56:02-0800 [-] Main loop terminated. 2013-12-09 10:56:02-0800 [-] Server Shut Down. I rebooted the slave and it connected to the master fine. We'll see if that unsticks the reconfig.
It didn't seem to unstick (not quickly enough for me, at least), so I initiated a second reconfig. Waiting for that now.
Didn't help. I had to graceful shutdown the master.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2835]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2835] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2841]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2841] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2843]
No activity + TC!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.