1438152 - Tree closure: windows builds in taskcluster not starting

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

+++ This bug was initially created as a clone of Bug #1372172 +++

Fom bug 1372172 comment 24:

Trees are closing again because Windows builds stopped running after some time. Do we hit an AWS query limit? We had increased the max capacities, hit a rate limit, decreased it but are still at 2x the default.

https://tools.taskcluster.net/aws-provisioner/gecko-3-b-win2012/ currently lists 256 pending jobs for gecko-3-b-win2012.

Pending build jobs whose platforms start with 'win' grouped by state: https://sql.telemetry.mozilla.org/queries/51150
pending	394
running	5
The 'running' number is falling.

Pending build jobs whose platforms start with 'win', looked at the column 'state': https://sql.telemetry.mozilla.org/queries/51149

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

turned out the problem was the same as this morning.

the ssh key associated with windows instances changed its naming convention to no longer include the workerType name in the ssh key name.

occ required patching in two places to no longer obtain the worker type from the ssh key name but instead from userdata on provisioned instances.

this mornings patch was:
https://github.com/mozilla-releng/OpenCloudConfig/commit/d7d2df5a174087bad52e7d3636ae92e043f999f0

and this afternoons patch is:
https://github.com/mozilla-releng/OpenCloudConfig/commit/137c8c1b0e4b3927f15cf38ee4f9771894818221

we will need to manually terminate windows instances close to the time that these jobs are nearing completion:
https://tools.taskcluster.net/groups/cBf4vjVTTrmTaZHrAXQhJA

in order to allow the new amis to come into use.