Buildbot Shutdown stalled by Pulse

RESOLVED INCOMPLETE

Status

RESOLVED INCOMPLETE
3 years ago
6 months ago

People

(Reporter: Callek, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
On bm110 today, we had the graceful-restart script running and for bm110, there were no errors.

On the slave page: http://buildbot-master110.bb.releng.scl3.mozilla.com:8201/buildslaves?no_builders=1 there was nothing listed as connected.

And the following was in the twistd.log

2016-02-14 03:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 08:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds



I saved logs as  `tar -cjvf ~cltbld/valentines.twistd.log.tar.xz master/twistd.*` for posterity
(Reporter)

Comment 1

3 years ago
Same issue on bm54 (did not save twistd.log there)
(Reporter)

Comment 2

3 years ago
And again on bm72 -- only this time its also claiming a buildrequest with no connected slaves:

(in a rough loop)
2016-02-14 07:50:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:00:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:00:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:00:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:10:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:15:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:15:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:20:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:30:46-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:30:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:30:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds

Also did not save the log though.
That's perfectly normal behavior. There must have been something else preventing the master from shutting down.

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INCOMPLETE
(Assignee)

Updated

6 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.