Open Bug 1422154 Opened 4 years ago Updated 4 years ago

Transition to modern EC2 instance types

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(Not tracked)

People

(Reporter: gps, Unassigned)

References

(Blocks 1 open bug)

Details

Many of the AWS workers (notably testers) are still running c3, m3, and r3 instance types.

The newer instance types will be faster and cheaper.

Drawbacks to upgrading the instance types:

* We lose ephemeral storage and will have to use EBS
* Potential implications for rate limiting / capacity limits due to increased EBS usage
* Timings will change and we may start seeing new test failures
* Other things may change causing test fallout

I know others have looked at this in the past. I believe there were legitimate reasons for holding off on the transition. But now that the c5 and m5 instance types are out, we're a full two generations behind. We should get with the times our infrastructure is modern and we're not dealing with a fire drill when AWS decides to drop the legacy instance types.
(In reply to Gregory Szorc [:gps] from comment #0)
> Many of the AWS workers (notably testers) are still running c3, m3, and r3
> instance types.
> 
> The newer instance types will be faster and cheaper.
> 
> Drawbacks to upgrading the instance types:
> 
> * We lose ephemeral storage and will have to use EBS
> * Potential implications for rate limiting / capacity limits due to
> increased EBS usage

There are ways to mitigate this risk.  AWS support first can help understand how close we're coming to the limits.  I think trusted advisor in our account can also help understand how close we are.  

Outside of that, there is also a service that can be deployed for the AWS account to monitor our limits and how close we are in realtime.  I only mention all of this because hitting EBS limits has been a very real thing in the not-so-distant past and it's easy to not have it setup right.
keep in mind we just switched in September away from m1.medium- that took a long time to green up tests
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.