Closed Bug 1221197 Opened 10 years ago Closed 10 years ago

Shift TRY load from b-2008-ix to y-2008-spot

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: grenade, Assigned: grenade)

References

Details

Over the next few hours (2015-11-03 09:30 PT onward) we'll be shifting TRY load from b-2008-ix instances to y-2008-spot instances in EC2. If you experience problems with your Windows TRY builds, please ping markco or grenade in #releng This bug will track the running and paused instances. IX instances will only be disabled in SlaveAlloc and can be re-enabled if we run into problems. Similarly spot instances can be easily disabled in SlaveAlloc if they are causing problems.
all b-2008-ix TRY instances disabled in slavealloc: update slaves set enabled=0 where name like 'b-2008-ix-%' and trustid=4; 10 on-demand instances enabled and spun up (5 in use1, 5 in usw2): update slaves set locked_masterid = null, enabled = 1 where name in ('y-2008-ec2-0001', 'y-2008-ec2-0002', 'y-2008-ec2-0003', 'y-2008-ec2-0004', 'y-2008-ec2-0005', 'y-2008-ec2-0016', 'y-2008-ec2-0017', 'y-2008-ec2-0018', 'y-2008-ec2-0019', 'y-2008-ec2-0020'); 9 spot instances enabled (4 in use1, 5 in usw2): update slaves set locked_masterid = null, enabled = 1 where name in ('y-2008-spot-001', 'y-2008-spot-003', 'y-2008-spot-004', 'y-2008-spot-005', 'y-2008-spot-006', 'y-2008-spot-007', 'y-2008-spot-008', 'y-2008-spot-009', 'y-2008-spot-010', 'y-2008-spot-101', 'y-2008-spot-102', 'y-2008-spot-103', 'y-2008-spot-104', 'y-2008-spot-105', 'y-2008-spot-106', 'y-2008-spot-107', 'y-2008-spot-108', 'y-2008-spot-109', 'y-2008-spot-110');
comment above should have read "19 spot"
There was an issue in userdata causing a bootloop on new instances. Now corrected with https://github.com/mozilla/build-cloud-tools/pull/128 I terminated all y-2008-spot and y-2008-ec2 instances and re-instantiated ec2. spot should come back on its own.
more spot instances enabled: mysql> select d.name dc, s.name slave from slaves s inner join datacenters d on d.dcid = s.dcid where enabled = 1 and locked_masterid is null and s.name like 'y-2008-spot-%' order by d.name, s.name; +-----------+-----------------+ | dc | slave | +-----------+-----------------+ | us-east-1 | y-2008-spot-001 | | us-east-1 | y-2008-spot-004 | | us-east-1 | y-2008-spot-005 | | us-east-1 | y-2008-spot-006 | | us-east-1 | y-2008-spot-007 | | us-east-1 | y-2008-spot-008 | | us-east-1 | y-2008-spot-009 | | us-east-1 | y-2008-spot-010 | | us-east-1 | y-2008-spot-011 | | us-east-1 | y-2008-spot-012 | | us-east-1 | y-2008-spot-013 | | us-east-1 | y-2008-spot-014 | | us-east-1 | y-2008-spot-015 | | us-east-1 | y-2008-spot-016 | | us-east-1 | y-2008-spot-017 | | us-east-1 | y-2008-spot-018 | | us-east-1 | y-2008-spot-019 | | us-east-1 | y-2008-spot-020 | | us-east-1 | y-2008-spot-021 | | us-east-1 | y-2008-spot-022 | | us-east-1 | y-2008-spot-023 | | us-east-1 | y-2008-spot-024 | | us-east-1 | y-2008-spot-025 | | us-east-1 | y-2008-spot-026 | | us-east-1 | y-2008-spot-027 | | us-east-1 | y-2008-spot-028 | | us-east-1 | y-2008-spot-029 | | us-east-1 | y-2008-spot-030 | | us-west-2 | y-2008-spot-101 | | us-west-2 | y-2008-spot-102 | | us-west-2 | y-2008-spot-103 | | us-west-2 | y-2008-spot-104 | | us-west-2 | y-2008-spot-105 | | us-west-2 | y-2008-spot-106 | | us-west-2 | y-2008-spot-107 | | us-west-2 | y-2008-spot-108 | | us-west-2 | y-2008-spot-109 | | us-west-2 | y-2008-spot-110 | | us-west-2 | y-2008-spot-111 | | us-west-2 | y-2008-spot-112 | | us-west-2 | y-2008-spot-113 | | us-west-2 | y-2008-spot-114 | | us-west-2 | y-2008-spot-115 | | us-west-2 | y-2008-spot-116 | | us-west-2 | y-2008-spot-117 | | us-west-2 | y-2008-spot-118 | | us-west-2 | y-2008-spot-119 | | us-west-2 | y-2008-spot-120 | | us-west-2 | y-2008-spot-121 | | us-west-2 | y-2008-spot-122 | | us-west-2 | y-2008-spot-123 | | us-west-2 | y-2008-spot-124 | | us-west-2 | y-2008-spot-125 | | us-west-2 | y-2008-spot-126 | | us-west-2 | y-2008-spot-127 | | us-west-2 | y-2008-spot-128 | | us-west-2 | y-2008-spot-129 | | us-west-2 | y-2008-spot-130 | +-----------+-----------------+ 58 rows in set (0.00 sec)
build results: mysql> select d.name dc, s.name, concat('http://', m.fqdn, ':', m.http_port, '/buildslaves/', s.name) slave from slaves s inner join datacenters d on d.dcid = s.dcid left outer join masters m on m.masterid = s.current_masterid where s.enabled = 1 and locked_masterid is null and (s.name like 'y-2008-spot-%' or s.name like 'y-2008-ec2-%') order by d.name, s.name; +-----------+-----------------+--------------------------------------------------------------------------------------+ | dc | name | slave | +-----------+-----------------+--------------------------------------------------------------------------------------+ | us-east-1 | y-2008-ec2-0001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0001 | | us-east-1 | y-2008-ec2-0002 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0002 | | us-east-1 | y-2008-ec2-0003 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0003 | | us-east-1 | y-2008-ec2-0004 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0004 | | us-east-1 | y-2008-ec2-0005 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0005 | | us-east-1 | y-2008-spot-001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-001 | | us-east-1 | y-2008-spot-004 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-004 | | us-east-1 | y-2008-spot-005 | NULL | | us-east-1 | y-2008-spot-006 | NULL | | us-east-1 | y-2008-spot-007 | NULL | | us-east-1 | y-2008-spot-008 | NULL | | us-east-1 | y-2008-spot-009 | NULL | | us-east-1 | y-2008-spot-010 | NULL | | us-east-1 | y-2008-spot-011 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-011 | | us-east-1 | y-2008-spot-012 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-012 | | us-east-1 | y-2008-spot-013 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-013 | | us-east-1 | y-2008-spot-014 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-014 | | us-east-1 | y-2008-spot-015 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-015 | | us-east-1 | y-2008-spot-016 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-016 | | us-east-1 | y-2008-spot-017 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-017 | | us-east-1 | y-2008-spot-018 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-018 | | us-east-1 | y-2008-spot-019 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-019 | | us-east-1 | y-2008-spot-020 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-020 | | us-east-1 | y-2008-spot-021 | NULL | | us-east-1 | y-2008-spot-022 | NULL | | us-east-1 | y-2008-spot-023 | NULL | | us-east-1 | y-2008-spot-024 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-024 | | us-east-1 | y-2008-spot-025 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-025 | | us-east-1 | y-2008-spot-026 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-026 | | us-east-1 | y-2008-spot-027 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-027 | | us-east-1 | y-2008-spot-028 | NULL | | us-east-1 | y-2008-spot-029 | NULL | | us-east-1 | y-2008-spot-030 | NULL | | us-west-2 | y-2008-ec2-0016 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0016 | | us-west-2 | y-2008-ec2-0017 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0017 | | us-west-2 | y-2008-ec2-0018 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0018 | | us-west-2 | y-2008-ec2-0019 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0019 | | us-west-2 | y-2008-ec2-0020 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0020 | | us-west-2 | y-2008-spot-101 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-101 | | us-west-2 | y-2008-spot-102 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-102 | | us-west-2 | y-2008-spot-103 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-103 | | us-west-2 | y-2008-spot-104 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-104 | | us-west-2 | y-2008-spot-105 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-105 | | us-west-2 | y-2008-spot-106 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-106 | | us-west-2 | y-2008-spot-107 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-107 | | us-west-2 | y-2008-spot-108 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-108 | | us-west-2 | y-2008-spot-109 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-109 | | us-west-2 | y-2008-spot-110 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-110 | | us-west-2 | y-2008-spot-111 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-111 | | us-west-2 | y-2008-spot-112 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-112 | | us-west-2 | y-2008-spot-113 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-113 | | us-west-2 | y-2008-spot-114 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-114 | | us-west-2 | y-2008-spot-115 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-115 | | us-west-2 | y-2008-spot-116 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-116 | | us-west-2 | y-2008-spot-117 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-117 | | us-west-2 | y-2008-spot-118 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-118 | | us-west-2 | y-2008-spot-119 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-119 | | us-west-2 | y-2008-spot-120 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-120 | | us-west-2 | y-2008-spot-121 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-121 | | us-west-2 | y-2008-spot-122 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-122 | | us-west-2 | y-2008-spot-123 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-123 | | us-west-2 | y-2008-spot-124 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-124 | | us-west-2 | y-2008-spot-125 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-125 | | us-west-2 | y-2008-spot-126 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-126 | | us-west-2 | y-2008-spot-127 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-127 | | us-west-2 | y-2008-spot-128 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-128 | | us-west-2 | y-2008-spot-129 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-129 | | us-west-2 | y-2008-spot-130 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-130 | +-----------+-----------------+--------------------------------------------------------------------------------------+ 68 rows in set (0.00 sec)
health: mysql> select d.name dc, s.name slave, concat('http://', m.fqdn, ':', m.http_port, '/buildslaves/', s.name) buildbot, concat('https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=', s.name) health from slaves s inner join datacenters d on d.dcid = s.dcid left outer join masters m on m.masterid = s.current_masterid where s.enabled = 1 and locked_masterid is null and (s.name like 'y-2008-spot-%' or s.name like 'y-2008-ec2-%') order by d.name, s.name; +-----------+-----------------+--------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+ | dc | slave | buildbot | health | +-----------+-----------------+--------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+ | us-east-1 | y-2008-ec2-0001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0001 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0001 | | us-east-1 | y-2008-ec2-0002 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0002 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0002 | | us-east-1 | y-2008-ec2-0003 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0003 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0003 | | us-east-1 | y-2008-ec2-0004 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0004 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0004 | | us-east-1 | y-2008-ec2-0005 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-ec2-0005 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0005 | | us-east-1 | y-2008-spot-001 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-001 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-001 | | us-east-1 | y-2008-spot-004 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-004 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-004 | | us-east-1 | y-2008-spot-005 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-005 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-005 | | us-east-1 | y-2008-spot-006 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-006 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-006 | | us-east-1 | y-2008-spot-007 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-007 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-007 | | us-east-1 | y-2008-spot-008 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-008 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-008 | | us-east-1 | y-2008-spot-009 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-009 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-009 | | us-east-1 | y-2008-spot-010 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-010 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-010 | | us-east-1 | y-2008-spot-011 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-011 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-011 | | us-east-1 | y-2008-spot-012 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-012 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-012 | | us-east-1 | y-2008-spot-013 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-013 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-013 | | us-east-1 | y-2008-spot-014 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-014 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-014 | | us-east-1 | y-2008-spot-015 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-015 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-015 | | us-east-1 | y-2008-spot-016 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-016 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-016 | | us-east-1 | y-2008-spot-017 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-017 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-017 | | us-east-1 | y-2008-spot-018 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-018 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-018 | | us-east-1 | y-2008-spot-019 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-019 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-019 | | us-east-1 | y-2008-spot-020 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-020 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-020 | | us-east-1 | y-2008-spot-021 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-021 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-021 | | us-east-1 | y-2008-spot-022 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-022 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-022 | | us-east-1 | y-2008-spot-023 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-023 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-023 | | us-east-1 | y-2008-spot-024 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-024 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-024 | | us-east-1 | y-2008-spot-025 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-025 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-025 | | us-east-1 | y-2008-spot-026 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-026 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-026 | | us-east-1 | y-2008-spot-027 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-027 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-027 | | us-east-1 | y-2008-spot-028 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-028 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-028 | | us-east-1 | y-2008-spot-029 | http://buildbot-master76.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-029 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-029 | | us-east-1 | y-2008-spot-030 | http://buildbot-master75.bb.releng.use1.mozilla.com:8101/buildslaves/y-2008-spot-030 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-030 | | us-west-2 | y-2008-ec2-0016 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0016 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0016 | | us-west-2 | y-2008-ec2-0017 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0017 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0017 | | us-west-2 | y-2008-ec2-0018 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0018 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0018 | | us-west-2 | y-2008-ec2-0019 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0019 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0019 | | us-west-2 | y-2008-ec2-0020 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-ec2-0020 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-ec2-0020 | | us-west-2 | y-2008-spot-101 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-101 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-101 | | us-west-2 | y-2008-spot-102 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-102 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-102 | | us-west-2 | y-2008-spot-103 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-103 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-103 | | us-west-2 | y-2008-spot-104 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-104 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-104 | | us-west-2 | y-2008-spot-105 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-105 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-105 | | us-west-2 | y-2008-spot-106 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-106 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-106 | | us-west-2 | y-2008-spot-107 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-107 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-107 | | us-west-2 | y-2008-spot-108 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-108 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-108 | | us-west-2 | y-2008-spot-109 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-109 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-109 | | us-west-2 | y-2008-spot-110 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-110 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-110 | | us-west-2 | y-2008-spot-111 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-111 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-111 | | us-west-2 | y-2008-spot-112 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-112 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-112 | | us-west-2 | y-2008-spot-113 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-113 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-113 | | us-west-2 | y-2008-spot-114 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-114 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-114 | | us-west-2 | y-2008-spot-115 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-115 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-115 | | us-west-2 | y-2008-spot-116 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-116 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-116 | | us-west-2 | y-2008-spot-117 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-117 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-117 | | us-west-2 | y-2008-spot-118 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-118 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-118 | | us-west-2 | y-2008-spot-119 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-119 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-119 | | us-west-2 | y-2008-spot-120 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-120 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-120 | | us-west-2 | y-2008-spot-121 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-121 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-121 | | us-west-2 | y-2008-spot-122 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-122 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-122 | | us-west-2 | y-2008-spot-123 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-123 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-123 | | us-west-2 | y-2008-spot-124 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-124 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-124 | | us-west-2 | y-2008-spot-125 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-125 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-125 | | us-west-2 | y-2008-spot-126 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-126 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-126 | | us-west-2 | y-2008-spot-127 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-127 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-127 | | us-west-2 | y-2008-spot-128 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-128 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-128 | | us-west-2 | y-2008-spot-129 | http://buildbot-master78.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-129 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-129 | | us-west-2 | y-2008-spot-130 | http://buildbot-master79.bb.releng.usw2.mozilla.com:8101/buildslaves/y-2008-spot-130 | https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=y-2008-spot-130 | +-----------+-----------------+--------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+ 68 rows in set (0.00 sec)
Load died right down this evening and most of these instances were sitting idle. Idleizer did not terminate or shut down anything so I initiated "graceful shutdown" manually on most instances which worked as it should and resulted in terminated instances for spot and stopped instances for ec2. When load rises again tomorrow, I expect cloud-tools to spin up spot instances as required.
i enabled a handful of ix machines (5) to take the first of the load in the morning in case spot is slow and i've not had coffee when the load hits.
re-enabled ix for the weekend as mark and i may be off grid for support. update slaves set enabled=1 where name like 'b-2008-ix-%' and trustid=4 and notes is null
disabled try spot for the weekend: update slaves set enabled=0 where name like 'y-2008-spot-%';
done.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.