Closed Bug 1249499 Opened 8 years ago Closed 8 years ago

Trees closed, Windows builds not starting

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Unassigned)

Details

Looks from https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=1a74a55e4d17&fromchange=d9c73b9a2a96 like it was sometime around 1:30pm Pacific today when we last started a Windows build.

Everything but Try is closed, though we aren't starting Windows builds on Try either.
I'm seeing this on aws-manager2 in aws_watch_pending.log :
Is g2.2xlarge the windows instance type?

42023 2016-02-18 18:36:14,626 - 7.67 (in us-west-2a) too expensive for g2.2xlarge
42024 2016-02-18 18:36:14,626 - 7.67 (in us-west-2c) too expensive for g2.2xlarge
42025 2016-02-18 18:36:14,626 - Ignoring AZ us-east-1e for g2.2xlarge becuase it is listed in  ignored      _azs: ['us-east-1b', 'us-east-1e']
instance type  vcpu   gb mem  savings over on-demand
g2.2xlarge 	 8	15	63%

frequency of being outbid / month
Medium

frequency of being outbid / week
Medium

We may need other contingencies for the windows boxen, for when we're outbid for a significant length of time.
We have slave which are up, but not running buildbot. Q and arr chased that down to powershell/userdata issue, and we've reverted part of a recent patch to resolve it:

https://github.com/mozilla/build-cloud-tools/commit/083245a9e9532152bd4e26cfffac0389113f94f7
The old instances have all been terminated, and we're just waiting for watch pending to launch some new ones. At the moment it's stuck on states like 'No slave name available for us-east-1, b-2008', which is a known issue with this system when you mass-terminate instances. They clear after about an hour. Filed bug 1249527 to investigate fixing that issue.
New instances are now starting to pick up work.
No more pending jobs for non-try windows builds, and try is much reduced.

Lets resolve this now, and leave it to the sheriffs to chose when to reopen the trees.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.