1142539 - [provisioner] make sure that we don't provision in only a single availability zone

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

When we provision, we shouldn't always pick the cheapest region.  We should do some sort of distribution across different regions, zones and instance types.  One idea is to count the number of instanes in each of these categories and weight them as less preferential when they have > 80% of of the load

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

Why? To avoid all nodes going away at once I guess.

Although that would be really great. Think of all the free hours Amazon would have given us :)
That said our best bet is probably to spread out, so we have fewer interruptions and bigger
chance of getting at least something killed (which is bonus).

By the way, as far as I remember, spot pricing is per availability-zone, not just region.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Yes, if we spawn all instances in a single zone/region, we're gone if that region goes out.  The idea here is that we don't want to only ever provision in the cheapest zone/region/type because if we do that we become really screwed if one goes offline.

Yep, spot pricing is by region and zone.  As of today, we now pick the cheapest type of the cheapest zone of the cheapest region.  The idea here is to pick the cheapest type, zone and region until we have, say, 80% of capacity in that one combination, then black list that combination to provision in other zones.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

This was resolved with the biaser.  We have a 5-10% price-comparison preference for region/zone/type combinations which don't have many instances.  We also bias (when influx is working) based on spot kills and provisioning failures to avoid bad regions.