1198317 - reduce the number of available b-2008-ix instances in TRY in order to force y-2008-spot instantiation

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

disabled instances:
b-2008-ix-0036
b-2008-ix-0039
b-2008-ix-0019
b-2008-ix-0038
b-2008-ix-0025
b-2008-ix-0054
b-2008-ix-0022
b-2008-ix-0058
b-2008-ix-0026
b-2008-ix-0059

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

disabled instances extended to include:

b-2008-ix-0030
b-2008-ix-0174
b-2008-ix-0057
b-2008-ix-0023
b-2008-ix-0047
b-2008-ix-0046
b-2008-ix-0041
b-2008-ix-0044
b-2008-ix-0061
b-2008-ix-0055
b-2008-ix-0043
b-2008-ix-0049
b-2008-ix-0035
b-2008-ix-0029
b-2008-ix-0031
b-2008-ix-0062
b-2008-ix-0045

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

all machines returned to pool.
will disable more tomorrow.

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

progress:
- reduced ix capacity to a single instance (b-2008-ix-0043)
- pushed win32, win64 m-c build to try (https://treeherder.mozilla.org/#/jobs?repo=try&revision=272cab1322fc)
- observed messages in watch pending log indicating our max bid price (0.4) would not be successful
- updated max bid price for y-2008 to 0.5 (https://github.com/mozilla/build-cloud-tools/pull/109)
- observed successful spot requests in ec2 console (3 for use1, 3 for usw2, as expected/configured in slavealloc)
- observed spot instances starting, successfully running userdata, naming themselves and mailing logs
- now awaiting build output at https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-272cab1322fc

Comment 4

[Rob Thijssen [:grenade (EET/UTC+0300)]]

us-east-1:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=try&name=y-2008-spot-001
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=try&name=y-2008-spot-002
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=try&name=y-2008-spot-003
us-west-2:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=try&name=y-2008-spot-101
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=try&name=y-2008-spot-102
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=try&name=y-2008-spot-103

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: spot-cltbld.png

the us-east-1 instances appear to have hung mid build. rdp'ing to the instances (001 - 003) as cltbld shows this running but apparently going nowhere cmd prompt.

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

the us-west-2 instances have all terminated. I cannot find any evidence that they did any work before terminating (slave_health/treeherder). The PaperTrail logs end like this:
Aug 27 02:09:48 y-2008-spot-101.try.releng.usw2.mozilla.com USER32:  The process c:\windows\SysWOW64\shutdown.exe (Y-2008-SPOT-101) has initiated the shutdown of computer Y-2008-SPOT-101 on behalf of user Y-2008-SPOT-101\cltbld for the following reason: No title for this reason could be found   Reason Code: 0x800000ff   Shutdown Type: shutdown   Comment: #015

Comment 7

[Rob Thijssen [:grenade (EET/UTC+0300)]]

I think we've demonstrated that the spinning up and terminating processes work. We obviously have work to do to get mozilla-build's undies untwisted, but that's another bug...

Comment 8

[Amy Rich [:arr] [:arich]]

There are alerts in #buildduty that indicate there's a buildbot misconfiguration/missing configuration:

[sns alert] Thu 06:08:03 PDT buildbot-master78.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 675 | First instance: 2015-08-27 05:28:27-0700 | Most recent instance: 2015-08-27 06:00:02-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.67.67
[sns alert] Thu 06:08:03 PDT buildbot-master78.bb.releng.usw2.mozilla.com watch_twistd_log.py: Count: 681 | First instance: 2015-08-27 05:28:27-0700 | Most recent instance: 2015-08-27 06:00:01-0700 | Twistd exception: twisted.cred.error.UnauthorizedLogin - unknown 10.132.67.101

I've verified that those are windows spot instances

10.132.67.67 (y-2008-spot-103) and 10.132.67.101 (y-2008-spot-102)

Comment 9

[Amy Rich [:arr] [:arich]]

All of the alerts were for use1 IPs, I didn't see any for usw2.