979564 - Create more spot instances

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

We're still overflowing onto on-demand instances for regular load, so let's create more spot nodes.

We currently have 200 tst-linux32-spot nodes allocated in each of usw2 and use1. We have 300 tst-linux64-spot nodes allocated in each region as well.

Let's create an additional 200 tst-linux32-spot and an extra 200 tst-linux64-spot nodes in each region.

TODO:
- Verify we have sufficient IP space
- Create network interfaces
- Create DNS entries
- Create slavealloc entries
- Adjust watch_pending configs

Comment 1

[Chris AtLee [:catlee]]

We don't have enough IP space in our 10.XXX.56.0/22 networks, will have to create some new subnets to put these into.

Comment 2

[Chris AtLee [:catlee]]

I created 10.XXX.156.0/22 in us-west-2 and us-east-1 for the extra spot nodes. I'm creating the additional interfaces now.

Comment 3

[Chris AtLee [:catlee]]

Bumped by 100 per type per region. Will do the other 100 tomorrow.

https://hg.mozilla.org/build/cloud-tools/rev/909db327f6f4

Comment 4

[Chris AtLee [:catlee]]

Created 100 more tst-linux{32,64}-spot nodes in both regions.

Comment 5

[Armen [:armenzg]]

Live.

Comment 6

[Chris AtLee [:catlee]]

I added the final tst-linux64 nodes, and fixed some issues in the spot bidding library (https://github.com/tarasglek/spotbidagent/commit/86ec660a452a6e0a446462c9c305b160be147129) and aws_watch_pending that were preventing us from using all the instances (https://hg.mozilla.org/build/cloud-tools/rev/879e7baced02). Now we're hitting our maximum limit of spot instances. I've emailed AWS to get that limit increased.