Closed Bug 1248257 Opened 8 years ago Closed 7 years ago

Buildbot Shutdown stalled by Pulse

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: Callek, Unassigned)

References

Details

On bm110 today, we had the graceful-restart script running and for bm110, there were no errors.

On the slave page: http://buildbot-master110.bb.releng.scl3.mozilla.com:8201/buildslaves?no_builders=1 there was nothing listed as connected.

And the following was in the twistd.log

2016-02-14 03:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 04:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 04:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 05:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 05:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 06:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 06:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:16:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:16:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:31:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:31:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 07:46:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 07:46:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:01:36-0800 [-] Pulse <0x79a92638>: heartbeat
2016-02-14 08:01:47-0800 [-] Pulse <0x79a92638>: Processed 1 events (1 heartbeats) in 0.00 seconds



I saved logs as  `tar -cjvf ~cltbld/valentines.twistd.log.tar.xz master/twistd.*` for posterity
Same issue on bm54 (did not save twistd.log there)
And again on bm72 -- only this time its also claiming a buildrequest with no connected slaves:

(in a rough loop)
2016-02-14 07:50:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:00:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:00:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:00:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:10:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:15:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:15:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds
2016-02-14 08:20:45-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:30:46-0800 [-] Claimed buildrequestids: [95489522L]
2016-02-14 08:30:48-0800 [-] Pulse <0x46d4560>: heartbeat
2016-02-14 08:30:59-0800 [-] Pulse <0x46d4560>: Processed 1 events (1 heartbeats) in 0.00 seconds

Also did not save the log though.
That's perfectly normal behavior. There must have been something else preventing the master from shutting down.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.