1315061 - Windows build jobs on XP and 8 x64 fail auto-retrys don't get scheduled

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

From #buildduty:
[sns alert] Nov 03 12:00:37 aws-manager2.srv.releng.scl3.mozilla.com aws_watch_pending.py: 2016-11-03 12:00:37,564 - No spot choices for b-2008

Windows build jobs on XP and 8 x64 fail (connection issues) and the auto-retrys don't get scheduled (at least don't show up on Treeherder). Previous trys failed up to 3 times.

Comment 1

[Michael Shal [:mshal]]

I think the problem is that Windows c4.2xlarge instances are bidding at over $8 / hr, and our bid price is $0.53. The $8/hr spike seems to only apply to this particular instance type, though. We might want to check a few different instance types (ie: fall back to a machine with lesser specs) to work around cases like this where prices are wacky for a specific machine.

Comment 2

[Chris AtLee [:catlee]]

We're definitely being outbid. Current prices are between $0.70 - $4.64/hr

Comment 3

[Chris AtLee [:catlee]]

Attachment: bump max bid to $1

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 8807277 [details] [review]
bump max bid to $1

coop reviewed and merged

Comment 5

[Chris AtLee [:catlee]]

starting to see bids for new instances go through.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Pending is dropping, based on $1 instances running in usw2a & usw2b.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: bump max bid to $1.30

We're seeing instances start and then be terminated for being outbid.

use1 is in crazy-land still - $4.64 in all availability zones we use. Prices in usw2 are high in 2c ($1.84), but in 2a and 2b it's bouncing around with a cap of 1.0.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8807356 [details] [review]
bump max bid to $1.30

Lets hold off on this, we're doing OK at the moment as prices in usw2a/2b go down a bit.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

All pending has gone. Pricing is recovering 
- us-east-1d is back to 41c/hr, 1a and 1c are still high. We could try enabling us-east-1e rather than excluding it
- us-west-2a and 2b are <= 42c/hr, 2a is still $1.84

Consequently I'm not seeing any swath of instances disappearing from being outbid, and in fact we're back to terminating them because of lack of work.

Go from us to reopen the trees.