Closed Bug 1221197 Opened 9 years ago Closed 9 years ago

Shift TRY load from b-2008-ix to y-2008-spot

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: grenade, Assigned: grenade)

References

Details

Over the next few hours (2015-11-03 09:30 PT onward) we'll be shifting TRY load from b-2008-ix instances to y-2008-spot instances in EC2. If you experience problems with your Windows TRY builds, please ping markco or grenade in #releng

This bug will track the running and paused instances.

IX instances will only be disabled in SlaveAlloc and can be re-enabled if we run into problems.

Similarly spot instances can be easily disabled in SlaveAlloc if they are causing problems.
all b-2008-ix TRY instances disabled in slavealloc:
update slaves set enabled=0 where name like 'b-2008-ix-%' and trustid=4;

10 on-demand instances enabled and spun up (5 in use1, 5 in usw2):
update slaves set locked_masterid = null, enabled = 1 where name in ('y-2008-ec2-0001', 'y-2008-ec2-0002', 'y-2008-ec2-0003', 'y-2008-ec2-0004', 'y-2008-ec2-0005', 'y-2008-ec2-0016', 'y-2008-ec2-0017', 'y-2008-ec2-0018', 'y-2008-ec2-0019', 'y-2008-ec2-0020');

9 spot instances enabled (4 in use1, 5 in usw2):
update slaves set locked_masterid = null, enabled = 1 where name in ('y-2008-spot-001', 'y-2008-spot-003', 'y-2008-spot-004', 'y-2008-spot-005', 'y-2008-spot-006', 'y-2008-spot-007', 'y-2008-spot-008', 'y-2008-spot-009', 'y-2008-spot-010', 'y-2008-spot-101', 'y-2008-spot-102', 'y-2008-spot-103', 'y-2008-spot-104', 'y-2008-spot-105', 'y-2008-spot-106', 'y-2008-spot-107', 'y-2008-spot-108', 'y-2008-spot-109', 'y-2008-spot-110');
comment above should have read "19 spot"
There was an issue in userdata causing a bootloop on new instances. Now corrected with https://github.com/mozilla/build-cloud-tools/pull/128
I terminated all y-2008-spot and y-2008-ec2 instances and re-instantiated ec2. spot should come back on its own.
more spot instances enabled:

mysql> select d.name dc, s.name slave from slaves s inner join datacenters d on d.dcid = s.dcid where enabled = 1 and locked_masterid is null and s.name like 'y-2008-spot-%' order by d.name, s.name;
+-----------+-----------------+
| dc        | slave           |
+-----------+-----------------+
| us-east-1 | y-2008-spot-001 |
| us-east-1 | y-2008-spot-004 |
| us-east-1 | y-2008-spot-005 |
| us-east-1 | y-2008-spot-006 |
| us-east-1 | y-2008-spot-007 |
| us-east-1 | y-2008-spot-008 |
| us-east-1 | y-2008-spot-009 |
| us-east-1 | y-2008-spot-010 |
| us-east-1 | y-2008-spot-011 |
| us-east-1 | y-2008-spot-012 |
| us-east-1 | y-2008-spot-013 |
| us-east-1 | y-2008-spot-014 |
| us-east-1 | y-2008-spot-015 |
| us-east-1 | y-2008-spot-016 |
| us-east-1 | y-2008-spot-017 |
| us-east-1 | y-2008-spot-018 |
| us-east-1 | y-2008-spot-019 |
| us-east-1 | y-2008-spot-020 |
| us-east-1 | y-2008-spot-021 |
| us-east-1 | y-2008-spot-022 |
| us-east-1 | y-2008-spot-023 |
| us-east-1 | y-2008-spot-024 |
| us-east-1 | y-2008-spot-025 |
| us-east-1 | y-2008-spot-026 |
| us-east-1 | y-2008-spot-027 |
| us-east-1 | y-2008-spot-028 |
| us-east-1 | y-2008-spot-029 |
| us-east-1 | y-2008-spot-030 |
| us-west-2 | y-2008-spot-101 |
| us-west-2 | y-2008-spot-102 |
| us-west-2 | y-2008-spot-103 |
| us-west-2 | y-2008-spot-104 |
| us-west-2 | y-2008-spot-105 |
| us-west-2 | y-2008-spot-106 |
| us-west-2 | y-2008-spot-107 |
| us-west-2 | y-2008-spot-108 |
| us-west-2 | y-2008-spot-109 |
| us-west-2 | y-2008-spot-110 |
| us-west-2 | y-2008-spot-111 |
| us-west-2 | y-2008-spot-112 |
| us-west-2 | y-2008-spot-113 |
| us-west-2 | y-2008-spot-114 |
| us-west-2 | y-2008-spot-115 |
| us-west-2 | y-2008-spot-116 |
| us-west-2 | y-2008-spot-117 |
| us-west-2 | y-2008-spot-118 |
| us-west-2 | y-2008-spot-119 |
| us-west-2 | y-2008-spot-120 |
| us-west-2 | y-2008-spot-121 |
| us-west-2 | y-2008-spot-122 |
| us-west-2 | y-2008-spot-123 |
| us-west-2 | y-2008-spot-124 |
| us-west-2 | y-2008-spot-125 |
| us-west-2 | y-2008-spot-126 |
| us-west-2 | y-2008-spot-127 |
| us-west-2 | y-2008-spot-128 |
| us-west-2 | y-2008-spot-129 |
| us-west-2 | y-2008-spot-130 |
+-----------+-----------------+
58 rows in set (0.00 sec)
build results:

mysql> select d.name dc, s.name, concat('http://', m.fqdn, ':', m.http_port, '/buildslaves/', s.name) slave from slaves s inner join datacenters d on d.dcid = s.dcid left outer join masters m on m.masterid = s.current_masterid where s.enabled = 1 and locked_masterid is null and (s.name like 'y-2008-spot-%' or s.name like 'y-2008-ec2-%') order by d.name, s.name;
+-----------+-----------------+--------------------------------------------------------------------------------------+
| dc        | name            | slave                                                                                |
+-----------+-----------------+--------------------------------------------------------------------------------------+
| us-east-1 | y-2008-ec2-0001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0001 |
| us-east-1 | y-2008-ec2-0002 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0002 |
| us-east-1 | y-2008-ec2-0003 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0003 |
| us-east-1 | y-2008-ec2-0004 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0004 |
| us-east-1 | y-2008-ec2-0005 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0005 |
| us-east-1 | y-2008-spot-001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-001 |
| us-east-1 | y-2008-spot-004 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-004 |
| us-east-1 | y-2008-spot-005 | NULL                                                                                 |
| us-east-1 | y-2008-spot-006 | NULL                                                                                 |
| us-east-1 | y-2008-spot-007 | NULL                                                                                 |
| us-east-1 | y-2008-spot-008 | NULL                                                                                 |
| us-east-1 | y-2008-spot-009 | NULL                                                                                 |
| us-east-1 | y-2008-spot-010 | NULL                                                                                 |
| us-east-1 | y-2008-spot-011 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-011 |
| us-east-1 | y-2008-spot-012 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-012 |
| us-east-1 | y-2008-spot-013 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-013 |
| us-east-1 | y-2008-spot-014 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-014 |
| us-east-1 | y-2008-spot-015 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-015 |
| us-east-1 | y-2008-spot-016 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-016 |
| us-east-1 | y-2008-spot-017 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-017 |
| us-east-1 | y-2008-spot-018 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-018 |
| us-east-1 | y-2008-spot-019 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-019 |
| us-east-1 | y-2008-spot-020 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-020 |
| us-east-1 | y-2008-spot-021 | NULL                                                                                 |
| us-east-1 | y-2008-spot-022 | NULL                                                                                 |
| us-east-1 | y-2008-spot-023 | NULL                                                                                 |
| us-east-1 | y-2008-spot-024 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-024 |
| us-east-1 | y-2008-spot-025 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-025 |
| us-east-1 | y-2008-spot-026 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-026 |
| us-east-1 | y-2008-spot-027 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-027 |
| us-east-1 | y-2008-spot-028 | NULL                                                                                 |
| us-east-1 | y-2008-spot-029 | NULL                                                                                 |
| us-east-1 | y-2008-spot-030 | NULL                                                                                 |
| us-west-2 | y-2008-ec2-0016 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0016 |
| us-west-2 | y-2008-ec2-0017 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0017 |
| us-west-2 | y-2008-ec2-0018 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0018 |
| us-west-2 | y-2008-ec2-0019 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0019 |
| us-west-2 | y-2008-ec2-0020 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0020 |
| us-west-2 | y-2008-spot-101 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-101 |
| us-west-2 | y-2008-spot-102 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-102 |
| us-west-2 | y-2008-spot-103 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-103 |
| us-west-2 | y-2008-spot-104 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-104 |
| us-west-2 | y-2008-spot-105 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-105 |
| us-west-2 | y-2008-spot-106 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-106 |
| us-west-2 | y-2008-spot-107 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-107 |
| us-west-2 | y-2008-spot-108 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-108 |
| us-west-2 | y-2008-spot-109 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-109 |
| us-west-2 | y-2008-spot-110 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-110 |
| us-west-2 | y-2008-spot-111 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-111 |
| us-west-2 | y-2008-spot-112 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-112 |
| us-west-2 | y-2008-spot-113 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-113 |
| us-west-2 | y-2008-spot-114 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-114 |
| us-west-2 | y-2008-spot-115 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-115 |
| us-west-2 | y-2008-spot-116 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-116 |
| us-west-2 | y-2008-spot-117 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-117 |
| us-west-2 | y-2008-spot-118 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-118 |
| us-west-2 | y-2008-spot-119 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-119 |
| us-west-2 | y-2008-spot-120 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-120 |
| us-west-2 | y-2008-spot-121 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-121 |
| us-west-2 | y-2008-spot-122 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-122 |
| us-west-2 | y-2008-spot-123 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-123 |
| us-west-2 | y-2008-spot-124 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-124 |
| us-west-2 | y-2008-spot-125 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-125 |
| us-west-2 | y-2008-spot-126 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-126 |
| us-west-2 | y-2008-spot-127 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-127 |
| us-west-2 | y-2008-spot-128 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-128 |
| us-west-2 | y-2008-spot-129 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-129 |
| us-west-2 | y-2008-spot-130 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-130 |
+-----------+-----------------+--------------------------------------------------------------------------------------+
68 rows in set (0.00 sec)
health:
mysql> select d.name dc, s.name slave, concat('http://', m.fqdn, ':', m.http_port, '/buildslaves/', s.name) buildbot, concat('https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=', s.name) health from slaves s inner join datacenters d on d.dcid = s.dcid left outer join masters m on m.masterid = s.current_masterid where s.enabled = 1 and locked_masterid is null and (s.name like 'y-2008-spot-%' or s.name like 'y-2008-ec2-%') order by d.name, s.name;
+-----------+-----------------+--------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
| dc        | slave           | buildbot                                                                             | health                                                                                              |
+-----------+-----------------+--------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
| us-east-1 | y-2008-ec2-0001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0001 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0001 |
| us-east-1 | y-2008-ec2-0002 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0002 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0002 |
| us-east-1 | y-2008-ec2-0003 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0003 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0003 |
| us-east-1 | y-2008-ec2-0004 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0004 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0004 |
| us-east-1 | y-2008-ec2-0005 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0005 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0005 |
| us-east-1 | y-2008-spot-001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-001 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-001 |
| us-east-1 | y-2008-spot-004 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-004 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-004 |
| us-east-1 | y-2008-spot-005 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-005 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-005 |
| us-east-1 | y-2008-spot-006 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-006 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-006 |
| us-east-1 | y-2008-spot-007 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-007 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-007 |
| us-east-1 | y-2008-spot-008 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-008 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-008 |
| us-east-1 | y-2008-spot-009 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-009 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-009 |
| us-east-1 | y-2008-spot-010 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-010 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-010 |
| us-east-1 | y-2008-spot-011 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-011 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-011 |
| us-east-1 | y-2008-spot-012 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-012 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-012 |
| us-east-1 | y-2008-spot-013 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-013 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-013 |
| us-east-1 | y-2008-spot-014 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-014 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-014 |
| us-east-1 | y-2008-spot-015 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-015 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-015 |
| us-east-1 | y-2008-spot-016 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-016 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-016 |
| us-east-1 | y-2008-spot-017 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-017 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-017 |
| us-east-1 | y-2008-spot-018 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-018 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-018 |
| us-east-1 | y-2008-spot-019 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-019 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-019 |
| us-east-1 | y-2008-spot-020 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-020 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-020 |
| us-east-1 | y-2008-spot-021 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-021 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-021 |
| us-east-1 | y-2008-spot-022 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-022 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-022 |
| us-east-1 | y-2008-spot-023 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-023 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-023 |
| us-east-1 | y-2008-spot-024 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-024 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-024 |
| us-east-1 | y-2008-spot-025 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-025 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-025 |
| us-east-1 | y-2008-spot-026 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-026 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-026 |
| us-east-1 | y-2008-spot-027 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-027 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-027 |
| us-east-1 | y-2008-spot-028 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-028 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-028 |
| us-east-1 | y-2008-spot-029 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-029 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-029 |
| us-east-1 | y-2008-spot-030 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-030 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-030 |
| us-west-2 | y-2008-ec2-0016 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0016 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0016 |
| us-west-2 | y-2008-ec2-0017 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0017 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0017 |
| us-west-2 | y-2008-ec2-0018 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0018 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0018 |
| us-west-2 | y-2008-ec2-0019 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0019 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0019 |
| us-west-2 | y-2008-ec2-0020 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0020 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0020 |
| us-west-2 | y-2008-spot-101 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-101 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-101 |
| us-west-2 | y-2008-spot-102 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-102 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-102 |
| us-west-2 | y-2008-spot-103 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-103 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-103 |
| us-west-2 | y-2008-spot-104 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-104 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-104 |
| us-west-2 | y-2008-spot-105 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-105 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-105 |
| us-west-2 | y-2008-spot-106 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-106 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-106 |
| us-west-2 | y-2008-spot-107 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-107 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-107 |
| us-west-2 | y-2008-spot-108 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-108 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-108 |
| us-west-2 | y-2008-spot-109 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-109 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-109 |
| us-west-2 | y-2008-spot-110 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-110 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-110 |
| us-west-2 | y-2008-spot-111 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-111 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-111 |
| us-west-2 | y-2008-spot-112 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-112 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-112 |
| us-west-2 | y-2008-spot-113 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-113 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-113 |
| us-west-2 | y-2008-spot-114 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-114 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-114 |
| us-west-2 | y-2008-spot-115 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-115 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-115 |
| us-west-2 | y-2008-spot-116 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-116 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-116 |
| us-west-2 | y-2008-spot-117 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-117 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-117 |
| us-west-2 | y-2008-spot-118 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-118 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-118 |
| us-west-2 | y-2008-spot-119 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-119 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-119 |
| us-west-2 | y-2008-spot-120 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-120 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-120 |
| us-west-2 | y-2008-spot-121 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-121 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-121 |
| us-west-2 | y-2008-spot-122 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-122 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-122 |
| us-west-2 | y-2008-spot-123 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-123 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-123 |
| us-west-2 | y-2008-spot-124 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-124 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-124 |
| us-west-2 | y-2008-spot-125 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-125 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-125 |
| us-west-2 | y-2008-spot-126 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-126 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-126 |
| us-west-2 | y-2008-spot-127 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-127 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-127 |
| us-west-2 | y-2008-spot-128 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-128 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-128 |
| us-west-2 | y-2008-spot-129 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-129 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-129 |
| us-west-2 | y-2008-spot-130 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-130 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-130 |
+-----------+-----------------+--------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
68 rows in set (0.00 sec)
Load died right down this evening and most of these instances were sitting idle. Idleizer did not terminate or shut down anything so I initiated "graceful shutdown" manually on most instances which worked as it should and resulted in terminated instances for spot and stopped instances for ec2.
When load rises again tomorrow, I expect cloud-tools to spin up spot instances as required.
i enabled a handful of ix machines (5) to take the first of the load in the morning in case spot is slow and i've not had coffee when the load hits.
re-enabled ix for the weekend as mark and i may be off grid for support.
update slaves set enabled=1 where name like 'b-2008-ix-%' and trustid=4 and notes is null
disabled try spot for the weekend:
update slaves set enabled=0 where name like 'y-2008-spot-%';
done.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.