Closed Bug 1047467 Opened 10 years ago Closed 10 years ago

c3.xlarge instance types are expensive, let's test running those tests on a range of instance types that are cheaper

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

x86
macOS
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kmoir, Assigned: kmoir)

Details

Attachments

(2 files, 1 obsolete file)

taras	kmoir: can you also set your new stuff
taras	to use a range of c3.xl, r3.xl and m3.xl?
	-->|	marcia (chatzilla@moz-E88C0D73.lightspeed.sntcca.sbcglobal.net) has joined #releng
	taras	just like we do for compiles?
	taras	or just set them to r3.xl for now
	taras	those are slightly cheape	
	catlee	yeah, I think http://hg.mozilla.org/build/cloud-tools/file/c10ba9d1f8b0/configs/watch_pending.cfg#l44 just needs adjusting
	catlee	kmoir: ^^ if you want to give that a shot
	kmoir	Right but have someone tested that the they all run on those instance types too?
	kmoir	without failure?
	taras	kmoir: i think that'll be you
	taras	r3.xl is nearly identical to c3.xl	
	kmoir	k. Well I'll open a bug and look at when I'm waiting for tests to complete on the other bug I'm working on for single locale builds for Android submitting to balrog
	catlee	it's a pretty easy config tweak
	kmoir	I know
	catlee	maybe try it on monday and see what happens
	kmoir	it's just the testing that takes time
	kmoir	would test on a branch first
	catlee	https://cheezburger.com/6615104768
Assignee: nobody → kmoir
Attached patch bug1047467.patchSplinter Review
Not sure how to test this on a branch without and not deploy to all.  I guess since it works for the builders it should work for the tests?
Attachment #8470091 - Flags: review?(catlee)
Comment on attachment 8470091 [details] [diff] [review]
bug1047467.patch

Review of attachment 8470091 [details] [diff] [review]:
-----------------------------------------------------------------

I think it's worth it to try and just keep a close eye on things. We can revert if necessary, and kill off the nodes if they're bad.

Please clean up the trailing whitespace.
Attachment #8470091 - Flags: review?(catlee) → review+
Attachment #8470091 - Flags: checked-in+
I haven't seen any issues with this on Tbpl since this was enable three hours ago and I can see in the aws console that the different instance types are being activated.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
coop|buildduty	should i be worried about the “400 Bad Request” errors in the aws_watch_pending.py output?
	nthomas	is there a lot of them ?
	nthomas	1 or a few is not unusual
	nthomas	haven't seen |Virtualization type 'hvm' is required for instances of type 'r3.xlarge'| before
	coop|buildduty	that’s the one
	coop|buildduty	repeated many, many times
	nthomas	http://hg.mozilla.org/build/cloud-tools/rev/3607eb84d963 seems unrelated
	nthomas	we can page rail or figure out what instances types to disable
	nthomas	I'm assuming AWS made a change which makes the API more restrictive
	nthomas	first message is Date: Mon, 11 Aug 2014 06:46:56 -0700
	nthomas	wonders why this hasn't been a big issue for launching build instances
	coop|buildduty	http://hg.mozilla.org/build/cloud-tools/rev/9dcb80cffe6c ?
	nthomas	bingo
	kmoir	yeah I landed that yesterday but it looked okay at the time
	kmoir	we do the same thing for the builders
	=-=	mtabara|lunch is now known as mtabara
	nthomas	there's probably something different about the ami though
	nthomas	I recall rail had a bunch of fun making one that supported both virtualisation types
	kmoir	so perhaps I should just remove r3.xlarge
	kmoir	yeah, I didn;t change the ami
	nthomas	https://aws.amazon.com/amazon-linux-ami/instance-type-matrix/ says m3.xlarge supports pv
	nthomas	so just removing r3.xlarge should be fine
	kmoir	okay I'll get a patch ready for review
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Attached patch bug1047467-2.patch (obsolete) — Splinter Review
Attachment #8471897 - Attachment is obsolete: true
Attachment #8471898 - Flags: review?(nthomas)
Attachment #8471898 - Flags: review?(nthomas) → review+
Attachment #8471898 - Flags: checked-in+
not a messsage in the log anymore
:rail how can I change the ami so it can support both virtualisation types?  r3.xlarge only supports HVM.  Looking at the amis in the aws console it seems you can only specify one type.
Flags: needinfo?(rail)
AMIs cannot support both virtualization types unfortunately... :(

We could add support for PV and HVM (using different AMIs), what would require:

* creating and testing HVM AMI for Ubuntu
* generate golden AMIs for both types
* add support for both AMI types in aws_watch_pending.py
Flags: needinfo?(rail)
Okay thanks rail, I'll investigate in bug 1053800
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.