1294584 - Provisioner is not able to spawn new desktop-test workers

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task)

Description

[Brian Stack [:bstack]]

desktop-test jobs are backed up and the trees are closed. We're investigating, and have found that new worker amis were likely misconfigured. The hvm version of the ami was added to a config meant to spawn m1.medium instances, which does not work. This should probably alert the operator sooner rather than later.

Related to bug 1294205.

Comment 1

[Greg Arndt [:garndt]]

Was this the bug created for the tree closure? If so, we should update this bug with details and either link this bug to something that will help mitigate this (possibly the ami set work) or a new bug to fix the problem elsewhere (maybe at least alerting when instance's can't be provisioned because of this reason).

Comment 2

[Brian Stack [:bstack]]

Retrospective: https://public.etherpad-mozilla.org/p/taskcluster-retro-2016-08-11

Copied below for posterity:

Retrospective: Provisioner and AMI Issues Aug 11, 2016
Bugzilla Bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1294584

Background:

    AMI updates in relation to bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1294205


Timeline:
    - ${timestamp?} <<< stuff that happens in that security bug that I can't see >>>
    - ${timestamp?} updates are made to AMIs and hvm type is put in pv-only worker def
    - 10:55:44 <gps> we have an image retrigger spiraling out of control
    - 14:03:46 <amiyaguchi> so judging from the aws-provisioner, taskcluster is pretty backed up today?
    - 16:26:54 we realize that the worker config for destktop-test is wrong and fix it
    - 17:00:00 still nothing is spawned in desktop-worker so we restart aws-provisioner
    - 17:26:00 there are new desktop-test workers spawned and we're back in action
    
Fallout:

    trees are closed after we get backed up to ~10,000 tasks pending


Action Items (should be achievable in short-medium term. immediately actionable):
    - this scenario should alert us somehow
    - maybe we can assert the virtualization type in worker-configs?


thanks:

    gps (again. perhaps we should call these retrospectives "gps reports")

    amiyaguchi