1354786 - taskcluster build jobs scheduled/pending for long time/not starting (build pool not scaling up?)

Status: RESOLVED FIXED

(Taskcluster :: Services, enhancement)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Trees are closed for this.

Taskcluster build jobs are pending but won't start (or take long, e.g. some Android build jobs are pending for 3+ hours). https://tools.taskcluster.net/aws-provisioner/ shows a small number of build jobs running and a big number pending. Jobs before 3:55am UTC push time have been completed. The one from 4:35am UTC still has pending jobs.

E.g. this jobs has still pending Android and OSX jobs:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=21062cd63dd72565a4f9afc35bb4f6f78726096c&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable

Comment 1

[Greg Arndt [:garndt]]

It appears the provisioner was stuck inside an iteration loop and never completed.  The loop started at this point:

https://papertrailapp.com/systems/taskcluster-aws-provisioner2/events?focus=787221693587431434&selected=787221693587431434

It performed some analysis, but then did not complete nor start up any subsequent iterations.  I restarted the app in heroku and I see the provisioner attempting to spawn instances now.

added jhford to the bug.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Trees reopened.