WEIRD: Builder.detached messages during reconfig

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
5 years ago
3 months ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2843] )

(Reporter)

Description

5 years ago
Got a ton of these which seems to be hanging the reconfig on bm54, all for the same slave (tst-linux64-ec2-368)
2013-12-09 10:56:04-0800 [Broker,77250,10.132.56.221] WEIRD: Builder.detached(<BuildSlave 'tst-linux64-ec2-368'>) (tst-linux64-ec2-368) not in attaching_slaves([]) or slaves([<SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-332'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-365'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-436'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-338'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-330'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-401'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-334'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-475'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-496'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-313'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-403'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-369'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-324'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-411'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-362'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-408'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-414'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-495'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-396'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-372'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-352'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-394'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-321'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-344'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-301'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-322'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-351'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-410'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-494'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-410'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-432'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-422'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-ec2-492'>, <SlaveBuilder builder='Ubuntu VM 12.04 x64 mozilla-inbound pgo test cppunit' slave='tst-linux64-spot-323'>])

The twistd.log on the slave looks normal, though I'm surprised the slave isn't running or even attempting to:
2013-12-09 10:56:02-0800 [Broker,client] lost remote
2013-12-09 10:56:02-0800 [Broker,client] lost remote
2013-12-09 10:56:02-0800 [Broker,client] lost remote
2013-12-09 10:56:02-0800 [Broker,client] lost remote
2013-12-09 10:56:02-0800 [Broker,client] lost remote
2013-12-09 10:56:02-0800 [Broker,client] lost remote
2013-12-09 10:56:02-0800 [Broker,client] Lost connection to buildbot-master54.srv.releng.usw2.mozilla.com:9201
2013-12-09 10:56:02-0800 [Broker,client] Stopping factory <buildslave.bot.BotFactory instance at 0x7f90f391eb00>
2013-12-09 10:56:02-0800 [-] Main loop terminated.
2013-12-09 10:56:02-0800 [-] Server Shut Down.


I rebooted the slave and it connected to the master fine. We'll see if that unsticks the reconfig.
(Reporter)

Comment 1

5 years ago
It didn't seem to unstick (not quickly enough for me, at least), so I initiated a second reconfig. Waiting for that now.
(Reporter)

Comment 2

5 years ago
Didn't help. I had to graceful shutdown the master.

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2835]

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2835] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2841]

Updated

4 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2841] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2843]
No activity + TC!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

3 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.