1248257 - Buildbot Shutdown stalled by Pulse

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Justin Wood (:Callek)]

On bm110 today, we had the graceful-restart script running and for bm110, there were no errors.

On the slave page: http://buildbot-master110.bb.releng.scl3.mozilla.com:8201/buildslaves?no_builders=1 there was nothing listed as connected.

And the following was in the twistd.log

2016-02-14 03:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 08:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds



I saved logs as  `tar -cjvf ~cltbld/valentines.twistd.log.tar.xz master/twistd.*` for posterity

Comment 1

[Justin Wood (:Callek)]

Same issue on bm54 (did not save twistd.log there)

Comment 2

[Justin Wood (:Callek)]

And again on bm72 -- only this time its also claiming a buildrequest with no connected slaves:

(in a rough loop)
2016-02-14 07:50:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:00:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:00:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:00:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:10:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:15:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:15:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:20:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:30:46-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:30:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:30:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds

Also did not save the log though.

Comment 3

[Chris AtLee [:catlee]]

That's perfectly normal behavior. There must have been something else preventing the master from shutting down.