1047467 - c3.xlarge instance types are expensive, let's test running those tests on a range of instance types that are cheaper

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

taras	kmoir: can you also set your new stuff
taras	to use a range of c3.xl, r3.xl and m3.xl?
	-->|	marcia ([e-mail]) has joined #releng
	taras	just like we do for compiles?
	taras	or just set them to r3.xl for now
	taras	those are slightly cheape	
	catlee	yeah, I think http://hg.mozilla.org/build/cloud-tools/file/c10ba9d1f8b0/configs/watch_pending.cfg#l44 just needs adjusting
	catlee	kmoir: ^^ if you want to give that a shot
	kmoir	Right but have someone tested that the they all run on those instance types too?
	kmoir	without failure?
	taras	kmoir: i think that'll be you
	taras	r3.xl is nearly identical to c3.xl	
	kmoir	k. Well I'll open a bug and look at when I'm waiting for tests to complete on the other bug I'm working on for single locale builds for Android submitting to balrog
	catlee	it's a pretty easy config tweak
	kmoir	I know
	catlee	maybe try it on monday and see what happens
	kmoir	it's just the testing that takes time
	kmoir	would test on a branch first
	catlee	https://cheezburger.com/6615104768

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1047467.patch

Not sure how to test this on a branch without and not deploy to all.  I guess since it works for the builders it should work for the tests?

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 8470091 [details] [diff] [review]
bug1047467.patch

Review of attachment 8470091 [details] [diff] [review]:
-----------------------------------------------------------------

I think it's worth it to try and just keep a close eye on things. We can revert if necessary, and kill off the nodes if they're bad.

Please clean up the trailing whitespace.

Comment 3

[Kim Moir [:kmoir] ET]

Comment on attachment 8470091 [details] [diff] [review]
bug1047467.patch

http://hg.mozilla.org/build/cloud-tools/rev/9dcb80cffe6c

Comment 4

[Kim Moir [:kmoir] ET]

I haven't seen any issues with this on Tbpl since this was enable three hours ago and I can see in the aws console that the different instance types are being activated.

Comment 5

[Kim Moir [:kmoir] ET]

coop|buildduty	should i be worried about the “400 Bad Request” errors in the aws_watch_pending.py output?
	nthomas	is there a lot of them ?
	nthomas	1 or a few is not unusual
	nthomas	haven't seen |Virtualization type 'hvm' is required for instances of type 'r3.xlarge'| before
	coop|buildduty	that’s the one
	coop|buildduty	repeated many, many times
	nthomas	http://hg.mozilla.org/build/cloud-tools/rev/3607eb84d963 seems unrelated
	nthomas	we can page rail or figure out what instances types to disable
	nthomas	I'm assuming AWS made a change which makes the API more restrictive
	nthomas	first message is Date: Mon, 11 Aug 2014 06:46:56 -0700
	nthomas	wonders why this hasn't been a big issue for launching build instances
	coop|buildduty	http://hg.mozilla.org/build/cloud-tools/rev/9dcb80cffe6c ?
	nthomas	bingo
	kmoir	yeah I landed that yesterday but it looked okay at the time
	kmoir	we do the same thing for the builders
	=-=	mtabara|lunch is now known as mtabara
	nthomas	there's probably something different about the ami though
	nthomas	I recall rail had a bunch of fun making one that supported both virtualisation types
	kmoir	so perhaps I should just remove r3.xlarge
	kmoir	yeah, I didn;t change the ami
	nthomas	https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/ says m3.xlarge supports pv
	nthomas	so just removing r3.xlarge should be fine
	kmoir	okay I'll get a patch ready for review

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1047467-2.patch

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: bug1047467-2.patch

Comment 8

[Kim Moir [:kmoir] ET]

not a messsage in the log anymore

Comment 9

[Kim Moir [:kmoir] ET]

:rail how can I change the ami so it can support both virtualisation types?  r3.xlarge only supports HVM.  Looking at the amis in the aws console it seems you can only specify one type.

Comment 10

[Rail Aliiev [:rail]]

AMIs cannot support both virtualization types unfortunately... :(

We could add support for PV and HVM (using different AMIs), what would require:

* creating and testing HVM AMI for Ubuntu
* generate golden AMIs for both types
* add support for both AMI types in aws_watch_pending.py

Comment 11

[Kim Moir [:kmoir] ET]

Okay thanks rail, I'll investigate in bug 1053800