1219763 - check_ami should run more frequently via runner

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Kim Moir [:kmoir] ET]

We have many long standing spot instances in the tst-emulator64 class that were running out of disk space. This revealed a problem that was fixed in bug 1217863 to cleanup the problem directory. However, even though the new golden ami was deployed yesterday, we still have many instances of this class that are still running the old amis.  Looking at the runner logs, check_ami hasn't run in over a day.  Given the current load in this pool, we need to run check_ami more often so instances with old amis are recycled on at least a daily basis.

See tst-emulator64-spot-665.test.releng.use1.mozilla.com for an example

Comment 1

[Chris AtLee [:catlee]]

check_ami should be running each time runner runs...

Comment 2

[Kim Moir [:kmoir] ET]

So runner only runs on startup once the instance is instantiated?  I see several instances where the instance has been up for > 1 day and the last entry in the runner log is that buildbot is running.

Comment 3

[Chris AtLee [:catlee]]

runner runs between buildbot running.

buildbot should be stopping after each job, and having runner run again.

Comment 4

[Morgan Phillips [:mrrrgn]]

Merging the last idleizer patch changed == 'linux2' to if sys.platform == ('darwin', 'linux2') and not self._remote_shutdown_requested:

This is wrong, it was supposed to be if sys.platform in ('darwin', 'linux2') and not self._remote_shutdown_requested:

This prevents buildbot from shutting down.

So much fail.

Comment 5

[Morgan Phillips [:mrrrgn]]

Attachment: idlebug.diff

There was an uncaught bug that made it into the patch (looks like it happened in the merge after an 'hg rebase'). There was no test catch it, which is even worse. This adds a test and fixes the bug.

Comment 6

[Morgan Phillips [:mrrrgn]]

Last patch has reboots working again: 2015-11-02 09:14:42-0800 [Broker,client] Telling the master we want to shutdown after any running builds are finished
2015-11-02 09:14:42-0800 [Broker,client]  startCommand:shell [id 10780]
2015-11-02 09:14:42-0800 [Broker,client] RunProcess._startCommand
2015-11-02 09:14:42-0800 [Broker,client]  bash -c pwd

Can we mark this as resolved?

Comment 7

[Kim Moir [:kmoir] ET]

Thanks Morgan, looking at the aws console I can see in the console today that the AMIs are refreshed.