like http://docs.taskcluster.net/tools/task-inspector/#4S3BqPNSSJqg1W5S4cBuwQ/0 pending for 2 hours now :( and counting across trees especially but not only b2g-builds closing trees so that they can catch up and we look into it
Component: Buildduty → TaskCluster
Product: Release Engineering → Testing
QA Contact: bugspam.Callek
It appears that this is similar issues that we've been seeing lately with certain instance types not being available in the region we currently provision in. There are some steps (bandaids) we have been doing short term to decrease this issue (although not completely foolproof) and there are some longer term things we are doing. 1. John is deploying the new provisioner today and rolling it out for some tasks that are not currently reported to TH production to ensure that things work as we expect them. 2. With this new provisioner we can start a smarter strategy for obtaining these instances a. Availability Zones - Currently we request instances within a region and AWS picks the AZ...so far we've seen this only allocate within us-west-2c. Provisioner can deploy in diferrent AZ's to increase the pool of available instances b. Regions - Support for multi-region should be possible to increase not only the pool size, but also have redundancy and failover plans. Details around this need to be worked out because caching is only done in us-west. We would probably want the s3 buckets for artifacts and caches to live as close to the instance as possible 3. instance Types - our older provisioner will only provision a worker type within a single instance type. The new provisioner can have a lit of preferred instance types and try to get an instance using one of them, thus increasing our odds of getting an instance quicker. As of ~10am CDT, the trees were reopened because the backlog was looking much better.
trees reopen 8:03 since the backlog has cleared so marking fixed
Severity: blocker → normal
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Version: unspecified → Trunk
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.