1268668 - Lots of ondemand instances in us-west-2

Status: RESOLVED WORKSFORME

(Taskcluster :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We have at least one tutum host in each US region.  In us-west-2, we have quite a bit more, much of it running:

 * 2 tutum hosts
 * 1 EC2 Container Service host, active since january
 * 2 dockercloud from march (cloud-mirror, probably?)
 * 6 unnamed hosts with Windows AMIs
 * 1 "win2012-jmaher"

At a guess, we could kill the EC2 container service host and maybe some of the Windows hosts (I think we're provisioning Windows with AWS provisioner, right??)

Are we done with tutum?

Comment 1

[Pete Moore [:pmoore][:pete]]

The Windows hosts will mostly be mine and grenade's.

We're using them quite heavily at the moment. I typically have three different worker types active at a time, with up to 4 instances provisioned so that I can run 32 bit opt, 64 bit opt, 32 bit debug, 64 bit debug builds in parallel. It is still no doubt cheaper to have these instances running than it would be to work serially on fewer instances, as this would slow me down and thus also cost money.

Aside from the provisioner-spawned instances, there are the instances used to create the AMIs, which I keep running as I tend to hop on them to try things out.

I'm hoping to wrap this stuff up shortly, at which point I'll destroy any instances no longer needed.

jmaher had an instance too, as he was helping out.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Sounds good pete -- it might help to tag those instances with a "Name" that indicates you (jmaher's is already so tagged).  And of course the spot instances aren't a problem -- it's just the ondemand instances that might be left running and forgotten about that I want to monitor.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Any further action on this required? It sounds like all the instances are accounted for. 

If we're going to implement something to monitor this for us, let's move this into monitoring and change the bug title to reflect htat.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I don't think the tutum or EC2 Container Service hosts are accounted for.

Comment 5

[Greg Arndt [:garndt]]

Not sure why there are two tutum hosts, but one of those in us-west-2 should be the s3-copy-proxy

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Greg, I noticed a few ondemand surprises when I was looking at the AWS eye-watering issue.  It might be worth taking a spin through those in the console and reaching out to the owners?

Comment 7

[Greg Arndt [:garndt]]

Checked this again and it appears there are a few long running instances but they're related to some base ami generation for windows images, and docker cloud.  Things appear ok.