Closed Bug 984893 Opened 8 years ago Closed 8 years ago

Lazy long running instances doing nothing

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86_64
Linux
task
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: armenzg, Unassigned)

Details

e.g.:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=tst-linux64-spot&name=tst-linux64-spot-662
Not done a job since the 6th.
twistd.log says that the machine connects to the buildbot-master and then it reboots because it feels very idle.
2014-03-14 11:10:57-0700 [-] I feel very idle and was thinking of rebooting as soon as the buildmaster says it's OK
2014-03-14 11:10:57-0700 [-] Telling the master we want to shutdown after any running builds are finished

Loading this:
http://buildbot-master68.srv.releng.usw2.mozilla.com:8201/buildslaves/tst-linux64-spot-662
says "Slave will shut down gracefully when it is idle.".

[cltbld@tst-linux64-spot-662.test.releng.usw2.mozilla.com ~]$ uptime
 07:31:12 up 4 days,  3:23,  1 user,  load average: 0.00, 0.01, 0.05

The health page says that there's not been any reboots.

I will gracefully shut all of them down and terminate them.
We should investigate how to prevent that.

==== Lazy long running instances ====
0 tst-linux64-spot-662 (i-5d617b54, us-west-2): up for 97 hours (11d 15h:56m since last build)
1 tst-linux64-spot-400 (i-21fbe128, us-west-2): up for 89 hours (14d 19h:5m since last build)
2 tst-linux64-spot-325 (i-9afbe193, us-west-2): up for 89 hours (42d 14h:33m since last build)
3 tst-linux64-spot-464 (i-33d5bb3a, us-west-2): up for 88 hours (14d 19h:4m since last build)
4 tst-linux64-spot-311 (i-d38892da, us-west-2): up for 85 hours (34d 20h:36m since last build)
5 tst-linux64-spot-119 (i-48806d68, us-east-1): up for 33 hours (34d 20h:37m since last build)
6 tst-linux64-spot-196 (i-b6816c96, us-east-1): up for 33 hours (11d 16h:2m since last build)
I took care of them.
Component: Buildduty → Platform Support
QA Contact: armenzg → coop
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.