1309901 - Use spot instances for ad-hoc clusters

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Comment 1

[Frank Bertsch [:frank]]

Initially I'm going to just have all spots as TASK instances (they do not run the DHFS datanode daemon, which we don't use anyways - right?).

We could also consider having 1 Master, X number of next nodes be on-demand as well, and the final nodes (num_instances - X - 1) be spot instances. 

I'm going to set the initial bid be $.84, but this is configurable.

Comment 2

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] fbertsch:spots > mozilla:master

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

How much longer does it take on average for a cluster to bootstrap now? Note that cluster bootstrap times can afaik be highly volatile with spot instances. How do you plan to deal with that if users complain that clusters don't start up?

Comment 4

[Frank Bertsch [:frank]]

Unfortunately this is hard to test, because it is so variable. I've launched a few clusters and they all bootstrapped in about the same amount of time as I've seen before, e.g. "Waiting" status within 15 minutes. 

In general, I don't think we're going to run into an issue of getting kicked off machines or unable to start spot machines, since our bid is so high right now. However, we may run into periods where it happens. What if we have a config that we can use to switch off using spot clusters?

Comment 5

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Frank Bertsch [:frank] from comment #4)
> Unfortunately this is hard to test, because it is so variable. I've launched
> a few clusters and they all bootstrapped in about the same amount of time as
> I've seen before, e.g. "Waiting" status within 15 minutes. 
> 
> In general, I don't think we're going to run into an issue of getting kicked
> off machines or unable to start spot machines, since our bid is so high
> right now. However, we may run into periods where it happens. What if we
> have a config that we can use to switch off using spot clusters?

Yeah, that makes sense. It would be nice to have a checkbox to disable spot instances when launching a new cluster.

Comment 6

[Frank Bertsch [:frank]]

I was hesitant to let users decide not to use spot instances, since so many people consider their workloads the most important and may not understand that losing a spot instance does not mean failures for their jobs. I was thinking this would be more an admin thing, though I'm open to considering otherwise.

Comment 7

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Frank Bertsch [:frank] from comment #6)
> I was hesitant to let users decide not to use spot instances, since so many
> people consider their workloads the most important and may not understand
> that losing a spot instance does not mean failures for their jobs. I was
> thinking this would be more an admin thing, though I'm open to considering
> otherwise.

Agreed.

Comment 8

[Frank Bertsch [:frank]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #7)
> (In reply to Frank Bertsch [:frank] from comment #6)
> > I was hesitant to let users decide not to use spot instances, since so many
> > people consider their workloads the most important and may not understand
> > that losing a spot instance does not mean failures for their jobs. I was
> > thinking this would be more an admin thing, though I'm open to considering
> > otherwise.
> 
> Agreed.

Okay I'm going to implement it this way, with a switch just for admins.

Comment 9

[GitHub Autolander Bot]

Attachment: [telemetry-analysis-service] fbertsch:spots > mozilla:master