queue gecko-t-linux-large stopped provisioning new workers, no m3 instances available accoring to provisioner
Categories
(Taskcluster :: Operations and Service Requests, defect)
Tracking
(Not tracked)
People
(Reporter: aryx, Assigned: dhouse)
References
Details
Attachments
(1 file, 1 obsolete file)
|
573 bytes,
patch
|
tomprince
:
review+
|
Details | Diff | Splinter Review |
Trees are closed for this due to missing Linux x64 test coverage.
https://tools.taskcluster.net/provisioners/aws-provisioner-v1/worker-types?layout=table shows high pending counts for gecko-t-linux-large (4k) and gecko-t-win10-64-gpu (800). Active machines are in the state completed or also exceptions (win gpu).
Earlier this week this got fixed by restarting the provisioner/queue. Please do this again.
|
Reporter | |
Comment 1•6 years ago
|
||
Rob doesn't have the access to do this.
|
|
Reporter | |
Comment 2•6 years ago
|
||
Workers are getting provisioned again, reason unknown.
nthomas mentioned there might have been additional load: https://mozilla.logbot.info/ci/20190726#c16497512
|
|
Reporter | |
Comment 4•6 years ago
|
||
It was not a load issue, only 55 linux large instances were running and the provisioner showed them (mostly?) as having completed their last job but not taken anything new.
|
|
Reporter | |
Comment 5•6 years ago
|
||
The issue hit again (linux-large) around 3am UTC and went await at ~4:15am UTC. As it's the third time in a week, can this get investigated, please?
|
||
Comment 6•6 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #5)
The issue hit again (linux-large) around 3am UTC and went await at ~4:15am UTC. As it's the third time in a week, can this get investigated, please?
To be clear here, these are the gecko-t-linux-large workers that are failing to provision again?
There are a bunch of worker types that have linux-large and I want to make sure I can narrow my search.
|
|
||
Comment 7•6 years ago
|
||
My #1 suggestion here would be to switch away from the m3.large instances we're currently using for gecko-t-linux-large workers to the current generation of m5.large instances. I'm not sure if that's part of the problem here, but AWS certainly isn't adding any capacity in the m3-series, and the m5.large are only $0.02/hr.
I realize this would mean establishing a new baseline for tests. Let me know if that is palatable to you.
|
|
Reporter | |
Comment 8•6 years ago
|
||
Yes, on all 3 occurrences, at least linux-large lacked capacity.
Dave, please answer coop's question regarding new performance baselines caused by instance type changes to improve instance provisioning.
|
|
||
Comment 9•6 years ago
•
|
||
I found this in papertrail:
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.655Z INFO aws-provisioner-production: changeForType outcome (workerType=gecko-t-linux-large, pendingTasks=1199, runningCapacity=50, pendingCapacity=4, change=1190)
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.655Z INFO aws-provisioner-production: determined change (workerType=gecko-t-linux-large, change=1190)
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.657Z ERROR aws-provisioner-production: (workerType=gecko-t-linux-large, region=us-east-1, type=m3.large, zones=[])
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.658Z ERROR aws-provisioner-production: (workerType=gecko-t-linux-large, region=us-west-1, type=m3.large, zones=[])
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.658Z ERROR aws-provisioner-production: (workerType=gecko-t-linux-large, region=us-west-2, type=m3.large, zones=[])
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.659Z ERROR aws-provisioner-production: could not create any bid (workerType=gecko-t-linux-large, priceTrace=[])
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: 03:00:02.661Z ERROR aws-provisioner-production: error provisioning this worker type, skipping (workerType=gecko-t-linux-large, err={})
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: reportError - level: warning, tags: {"workerType":"gecko-t-linux-large"}
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: Error: Could not create any bid for gecko-t-linux-large
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: at subClass.WorkerType.determineSpotBids (/app/lib/worker-type.js:950:13)
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: at Provisioner.provision (/app/lib/provision.js:194:29)
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: at <anonymous>
Jul 28 23:00:02 taskcluster-aws-provisioner2 app/provisioner.1: at process._tickDomainCallback (internal/process/next_tick.js:228:7)
The error doesn't really tell me why it couldn't provision, but we're definitely hitting something here.
|
||
Comment 10•6 years ago
|
||
That error comes from
https://github.com/taskcluster/aws-provisioner/blob/c1a05150f2369069c6db5ab1470b5a3a7fa7be80/lib/worker-type.js#L889-L951
and in particular
if (pricingData[region] && pricingData[region][type]) {
zones = Object.keys(pricingData[region][type]);
}
The logging before the error indicates that there are zero zones in which this instance type is available.
So coop I think you've hit the nail on the head: m3's are disappearing. It's weird that it's intermittent, but AWS is mysterious. Maybe this is some kind of "gray-out"?
|
|
Reporter | |
Comment 11•6 years ago
|
||
This hit again (with linux-large): Pending queue started to grow at ~0:45am UTC, provisioning started at 03:10am UTC.
|
||
Comment 12•6 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)
Yes, on all 3 occurrences, at least linux-large lacked capacity.
Dave, please answer coop's question regarding new performance baselines caused by instance type changes to improve instance provisioning.
I can answer this in Dave's place: we don't see any problem with having new performance baselines. If we're getting a heads up when this happens, even better.
|
|
||
Comment 13•6 years ago
|
||
Note to whoever is making this change: m5's use EBS, as they do not have instance storage . So this isn't just s/m3/m5/g.
|
|
Reporter | |
Updated•6 years ago
|
|
|
Reporter | |
Comment 14•6 years ago
|
||
Issue is back, no new workers for the last 4h and a growing backlog since 2.5h.
|
|
||
Comment 15•6 years ago
|
||
Same issue, starting at:
Aug 03 15:36:11 taskcluster-aws-provisioner2 app/provisioner.1: 15:36:11.477Z ERROR aws-provisioner-production: could not create any bid (workerType=gecko-t-linux-large, priceTrace=[])
|
|
||
Updated•6 years ago
|
|
|
Assignee | |
Comment 17•6 years ago
|
||
Looks like I'll need to get a change in to https://hg.mozilla.org/ci/ci-configuration/file/tip/worker-pools.yml
the description in the workertype shows (https://tools.taskcluster.net/aws-provisioner/gecko-t-linux-large/view):
"description": "DO NOT EDIT - This resource is configured automatically by ci-admin.\n\n",
for the ebs disk difference, I'll review other workertypes to see if there are additional parameters needed or if the image needs changed
|
|
Assignee | |
Comment 18•6 years ago
|
||
installed ci-admin for checking the diff when I have it ready to commit:
$ hg clone https://hg.mozilla.org/ci/ci-admin/
$ cd ci-admin
$ pip3 install ./
[...]
$ cd ..
$ hg clone https://hg.mozilla.org/ci/ci-configuration/
$ cd ci-configuration
$ ci-admin diff --environment production --ci-configuration-directory ./ # environments defined at https://hg.mozilla.org/ci/ci-configuration/file/tip/environments.yml
--- current
+++ generated
@@ -81636,17 +81636,16 @@ Role=repo:github.com/mozilla-mobile/reference-browser:pull-request:
- project:mobile:reference-browser:releng:signing:format:*
- queue:cancel-task:mobile-level-1/*
- queue:create-task:highest:aws-provisioner-v1/mobile-1-*
- queue:create-task:highest:proj-autophone/gecko-t-ap-perf-g5
- queue:create-task:highest:proj-autophone/gecko-t-ap-perf-p2
- queue:create-task:highest:proj-autophone/gecko-t-bitbar-gw-perf-g5
- queue:create-task:highest:proj-autophone/gecko-t-bitbar-gw-perf-p2
- queue:create-task:highest:scriptworker-prov-v1/mobile-signing-dep-v1
- - queue:get-artifact:mobile/android-sdk/*
- queue:rerun-task:mobile-level-1/*
- queue:route:index.mobile.cache.level-1.*
- queue:route:index.mobile.v2.reference-browser.*
- queue:route:index.project.mobile.reference-browser.cache.level-1.*
- queue:route:index.project.mobile.reference-browser.staging-signed-nightly.*
- queue:route:index.project.mobile.reference-browser.v2.staging.*
- queue:route:notify.email.perftest-alerts@mozilla.com.on-failed
- queue:scheduler-id:mobile-level-1
|
|
Assignee | |
Comment 19•6 years ago
|
||
It looks like m3.large had a default instance store size of 32GB (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html).
For moving to ebs, we can specify the ebs size to match that at 32GB like:
diff --git a/worker-pools.yml b/worker-pools.yml
--- a/worker-pools.yml
+++ b/worker-pools.yml
@@ -1554,7 +1554,11 @@ aws-provisioner-v1/gecko-t-linux-large:
userData:
dockerConfig: {allowPrivileged: false}
instanceTypes:
- - instanceType: m3.large
+ - instanceType: m5.large
+ launchSpec:
+ BlockDeviceMappings:
+ - DeviceName: /dev/xvdb
+ Ebs: {DeleteOnTermination: true, VolumeSize: 32, VolumeType: gp2}
userData:
billingCycleInterval: 7200
capacityManagement: {diskspaceThreshold: 20000000000}
I need to verify the devicename is what is expected and that the volume type is correct.
|
|
Assignee | |
Comment 20•6 years ago
|
||
According to the aws migration doc, we can use the same ami: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html#resize-instance-store-backed-instance
|
|
Assignee | |
Comment 21•6 years ago
|
||
I still see this other potential change that I haven't tracked down yet:
$ ci-admin diff --environment production --ci-configuration-directory ./
[...]
@@ -81637,17 +81721,16 @@ Role=repo:github.com/mozilla-mobile/reference-browser:pull-request:
- project:mobile:reference-browser:releng:signing:format:*
- queue:cancel-task:mobile-level-1/*
- queue:create-task:highest:aws-provisioner-v1/mobile-1-*
- queue:create-task:highest:proj-autophone/gecko-t-ap-perf-g5
- queue:create-task:highest:proj-autophone/gecko-t-ap-perf-p2
- queue:create-task:highest:proj-autophone/gecko-t-bitbar-gw-perf-g5
- queue:create-task:highest:proj-autophone/gecko-t-bitbar-gw-perf-p2
- queue:create-task:highest:scriptworker-prov-v1/mobile-signing-dep-v1
- - queue:get-artifact:mobile/android-sdk/*
- queue:rerun-task:mobile-level-1/*
- queue:route:index.mobile.cache.level-1.*
- queue:route:index.mobile.v2.reference-browser.*
- queue:route:index.project.mobile.reference-browser.cache.level-1.*
- queue:route:index.project.mobile.reference-browser.staging-signed-nightly.*
- queue:route:index.project.mobile.reference-browser.v2.staging.*
- queue:route:notify.email.perftest-alerts@mozilla.com.on-failed
- queue:scheduler-id:mobile-level-1
home:ci-configuration house$ hg diff
diff --git a/worker-pools.yml b/worker-pools.yml
--- a/worker-pools.yml
+++ b/worker-pools.yml
@@ -1591,6 +1591,38 @@ aws-provisioner-v1/gecko-t-linux-large:
launchSpec: {SubnetId: subnet-2eaaba67}
- availabilityZone: us-west-2c
launchSpec: {SubnetId: subnet-540a9f0f}
+aws-provisioner-v1/gecko-t-linux-large-beta:
+ description: 'test move -large from m3 to m5'
+ owner: dhouse@mozilla.com
+ email_on_error: false
+ provider_id: legacy-aws-provisioner-v1
+ config:
+ image: docker-worker-hvm-builder-current
+ maxCapacity: 3500
+ scalingRatio: 0.1
+ userData:
+ dockerConfig: {allowPrivileged: false}
+ instanceTypes:
+ - instanceType: m5.large
+ launchSpec:
+ BlockDeviceMappings:
+ - DeviceName: /dev/xvdb
+ Ebs: {DeleteOnTermination: true, VolumeSize: 32, VolumeType: gp2}
+ userData:
+ billingCycleInterval: 7200
+ capacityManagement: {diskspaceThreshold: 20000000000}
+ regions:
+ - region: us-east-1
+ launchSpec:
+ SecurityGroupIds: [sg-12cd3762]
+ - region: us-west-2
+ launchSpec:
+ SecurityGroupIds: [sg-2728435d]
+ availabilityZones:
+ - availabilityZone: us-east-1a
+ launchSpec: {SubnetId: subnet-566e060c}
+ - availabilityZone: us-west-2b
+ launchSpec: {SubnetId: subnet-2eaaba67}
aws-provisioner-v1/gecko-t-linux-xlarge:
description: Worker for Firefox automation
owner: Firefox CI
It looks like it comes from grants.yml, but I'm not sure why it is showing as to be removed/dropped. Maybe it was added manually and not through ci-admin.
|
|
Assignee | |
Comment 22•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #21)
I still see this other potential change that I haven't tracked down yet:
[...]
@@ -81637,17 +81721,16 @@ Role=repo:github.com/mozilla-mobile/reference-browser:pull-request:
[...]
- queue:get-artifact:mobile/android-sdk/*
scope inspector shows it for the roleid:
RoleId
repo:github.com/mozilla-mobile/reference-browser:pull-request
Description
DO NOT EDIT - This resource is configured automatically by ci-admin.
Scopes in this role are defined in ci-configuration/grants.yml.
Created
6 months ago 2019-02-11T13:52:41.036Z
Last Modified
3 hours ago
It doesn't say who modified it.
I'll try adding that change to my local repo to avoid the risk of changing it and breaking something that someone is working on.
|
|
Assignee | |
Comment 23•6 years ago
|
||
testing with a -beta worker pool
|
|
Assignee | |
Comment 24•6 years ago
|
||
created fine? now need to figure out how to check the instances:
$ ci-admin apply --environment production --ci-configuration-directory ./
Creating AwsProvisionerWorkerType=gecko-t-linux-large-beta
|
|
Assignee | |
Comment 25•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #24)
created fine? now need to figure out how to check the instances:
$ ci-admin apply --environment production --ci-configuration-directory ./ Creating AwsProvisionerWorkerType=gecko-t-linux-large-beta
Looks like it was created correctly:
https://tools.taskcluster.net/aws-provisioner/gecko-t-linux-large-beta/
however it lists "No running instances" and for recent errors, "Errors not available".
Health (https://tools.taskcluster.net/aws-provisioner/aws-health) does not load after 2 minutes (maybe because there are no instances).
|
|
Assignee | |
Comment 26•6 years ago
|
||
I'm re-applying with a minCapacity=1 (and max 3 to keep failure thrashing low (?)). I'm expecting to see one worker come up and the health check to do something to show it is okay (or maybe I'll find logs for it somewhere).
|
|
Assignee | |
Comment 27•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #26)
I'm re-applying with a minCapacity=1 (and max 3 to keep failure thrashing low (?)). I'm expecting to see one worker come up and the health check to do something to show it is okay (or maybe I'll find logs for it somewhere).
No instances or pending instances are listed yet.
Maybe the bid pricing or region is wrong, but I would expect the provisioning errors to list that.
|
|
Assignee | |
Comment 28•6 years ago
|
||
m5.large are available in the region I am testing
I am testing changing the min and max bid pricing to 0.1 to 10 to see if the default bid pricing setting prevented the creation.
If that fails, I'll try switching to ondemand to verify with one minCapacity
|
|
Assignee | |
Comment 29•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #28)
m5.large are available in the region I am testing
I am testing changing the min and max bid pricing to 0.1 to 10 to see if the default bid pricing setting prevented the creation.
If that fails, I'll try switching to ondemand to verify with one minCapacity
it looks like the config gets replaced by what was set from my ci-admin applied ci-configuration values. So i'm testing with allowing ondemand from the config.
|
|
Assignee | |
Comment 30•6 years ago
|
||
That changed the results a little; worker type status now shows:
Instance Type Availability Zone Running Capacity Pending Capacity
m5.large us-east-1a 0 0
instead of the text "No running instances"
But the config stayed at canUseOndemand=false
I've removed the workertype and will try recreating it.
|
|
Assignee | |
Comment 31•6 years ago
|
||
I saw one instance go up for my testing -beta pool, but I think my removal and recreated broke that.
I'll check later if an instance is up (giving aws-provisioner enough time for a stable state), and then try applying to gecko-t-linux-large when the queue is lower.
|
|
Assignee | |
Comment 32•6 years ago
|
||
The gecko-t-linux-large queue is at 2k but trending down quickly. When it drops under 500, I'll change the workertype def to use m5 and see if all goes too well.
|
||
Comment 33•6 years ago
|
||
Note that when you create a worker, you'll need to manually create the corresponding secret (c.f. https://tools.taskcluster.net/secrets/worker-type%3Aaws-provisioner-v1%2Fgecko-t-linux-large)
|
|
||
Comment 34•6 years ago
|
||
|
|
Assignee | |
Comment 35•6 years ago
|
||
After various attempts poking at it, I terminated all of the gecko-t-linux-large instances. After that, the new/pending instances came up as m5.large. I"ll have to wait and see if they complete tasks without problems.
|
|
Assignee | |
Comment 36•6 years ago
|
||
(In reply to Tom Prince [:tomprince] from comment #34)
Comment on attachment 9083147 [details] [diff] [review]
patchReview of attachment 9083147 [details] [diff] [review]:
This looks reasonable for testing. When it lands, both the
gecko-t-linux-large and gecko-3-t-linux-large workers should be adjusted.
I didn't find a "gecko-3-t-linux-large" worker type, but only a "gecko-3-t-linux-xlarge". Am I missing something, or is there a different workertype that needs adjusted?
|
|
Assignee | |
Comment 37•6 years ago
|
||
Thanks Tom, here matches your earlier feedback (removing the current state scope for mobile). I've already applied this.
|
|
Assignee | |
Comment 38•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #35)
After various attempts poking at it, I terminated all of the gecko-t-linux-large instances. After that, the new/pending instances came up as m5.large. I"ll have to wait and see if they complete tasks without problems.
The first group of tasks that were running through were cancelled by the user.
More are going through now and a few have completed. From the logs for the few I've spot checked, it looks like things are working as expected.
|
|
Assignee | |
Comment 39•6 years ago
|
||
(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #12)
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)
Yes, on all 3 occurrences, at least linux-large lacked capacity.
Dave, please answer coop's question regarding new performance baselines caused by instance type changes to improve instance provisioning.
I can answer this in Dave's place: we don't see any problem with having new performance baselines. If we're getting a heads up when this happens, even better.
I switched gecko-t-linux-large over from m3 to m5 at 00:05 pacific Tues Aug 6 (about 35 minutes ago).
And as a future warning, from Tom's note earlier it looks like there may be another worker type that needs to be changed. I'll let you know if I change another and when.
|
|
||
Comment 40•6 years ago
|
||
Nice work! A few notes about AWS provisioner below, all of which should prepare you to be delighted by the behavior of worker-manager :)
however it lists "No running instances" and for recent errors, "Errors not available".
Health (https://tools.taskcluster.net/aws-provisioner/aws-health) does not load after 2 minutes (maybe because there are no instances).
This is pretty normal -- the health / errors view (which I think are the same?) doesn't generally show much of use.
I am testing changing the min and max bid pricing to 0.1 to 10 to see if the default bid pricing setting prevented the creation.
If that fails, I'll try switching to ondemand to verify with one minCapacity
Since at this point we just get the current spot price regardless of bids (AWS's pricing model isn't a "market" anymore), prices are largely irrelevant, which is why maxPrice is so high for everything. The canUseOndemand and canUseSpot options have always been ignored.
|
|
Assignee | |
Comment 41•6 years ago
|
||
The gecko-t-linux-large worker(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #40)
Nice work! A few notes about AWS provisioner below, all of which should prepare you to be delighted by the behavior of worker-manager :)
however it lists "No running instances" and for recent errors, "Errors not available".
Health (https://tools.taskcluster.net/aws-provisioner/aws-health) does not load after 2 minutes (maybe because there are no instances).This is pretty normal -- the health / errors view (which I think are the same?) doesn't generally show much of use.
I am testing changing the min and max bid pricing to 0.1 to 10 to see if the default bid pricing setting prevented the creation.
If that fails, I'll try switching to ondemand to verify with one minCapacitySince at this point we just get the current spot price regardless of bids (AWS's pricing model isn't a "market" anymore), prices are largely irrelevant, which is why maxPrice is so high for everything. The canUseOndemand and canUseSpot options have always been ignored.
Thanks! That explains the behavior and numbers I didn't understand. I'll look forward to the worker-manager.
|
|
||
Updated•6 years ago
|
|
|
||
Comment 42•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #36)
gecko-t-linux-large and gecko-3-t-linux-large workers should be adjusted.
I didn't find a "gecko-3-t-linux-large" worker type, but only a "gecko-3-t-linux-xlarge". Am I missing something, or is there a different workertype that needs adjusted?
That was the worker-type I was thinking of. I see it uses m3.xlarge not m3.large, and also c3.xlarge. I guess one or both those don't have availability issues we are struggling with.
|
|
||
Updated•6 years ago
|
|
|
Assignee | |
Comment 43•6 years ago
|
||
Since the change, I see many "Error requesting new instance" messages for us-west-1c and some for us-west-1b.
|
|
Assignee | |
Comment 44•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #43)
Since the change, I see many "Error requesting new instance" messages for us-west-1c and some for us-west-1b.
I think wander restarted the aws-provider yesterday and it then provisioned instances more quickly
|
|
Assignee | |
Comment 45•6 years ago
|
||
:aryx, does the gecko-t-linux-large pool look okay after this change? I see a lot of exceptions in the metrics; is that accurate? (from treeherder/actuals)
|
|
Assignee | |
Comment 46•6 years ago
|
||
|
|
Reporter | |
Comment 47•6 years ago
|
||
There is an elevated exception level which started earlier today (European morning). It's not a critical level where it would be a threat to tree status.
|
|
Assignee | |
Comment 48•6 years ago
|
||
|
|
Assignee | |
Comment 49•6 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #47)
There is an elevated exception level which started earlier today (European morning). It's not a critical level where it would be a threat to tree status.
Re: discussion in irc and https://bugzilla.mozilla.org/show_bug.cgi?id=1572393#c23 , the exceptions may be caused by spot instances being reclaimed by aws because of ondemand demand
We have not seen a period with zero spot requests fulfilled on the new m5's for gecko-t-linux-large. So I'll close this bug out as fixed.
|
|
||
Comment 50•6 years ago
|
||
(In reply to Dave House [:dhouse] from comment #39)
(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #12)
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)
Yes, on all 3 occurrences, at least linux-large lacked capacity.
Dave, please answer coop's question regarding new performance baselines caused by instance type changes to improve instance provisioning.
I can answer this in Dave's place: we don't see any problem with having new performance baselines. If we're getting a heads up when this happens, even better.
I switched gecko-t-linux-large over from m3 to m5 at 00:05 pacific Tues Aug 6 (about 35 minutes ago).
And as a future warning, from Tom's note earlier it looks like there may be another worker type that needs to be changed. I'll let you know if I change another and when.
Thanks!
Description
•