Closed Bug 1599642 Opened 6 years ago Closed 6 years ago

Generate experiments_v1 in bigquery-etl or configure the dataproc job to autoscale

Categories

(Data Platform and Tools :: General, enhancement)

Product:
Data Platform and Tools
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: relud, Assigned: relud)

Details

I think it will be relatively easy to migrate experiments_v1 to bigquery-etl, so long as we either:

  • set it up to depend on a new job to load the experiment list as a reference table
    or
  • set it up to run from a script that passes the experiment list as a parameter

However if something comes up that pushes that timeline out past the end of next week, we should configure the dataproc job to autoscale, because 40 n1-highmem-8 instances is overkill when the experiment population is low.

The reason the dataproc job does not autoscale right now is because our airflow configuration doesn't yet have support for it.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Datasets: Experiments → General
You need to log in before you can comment on or make changes to this bug.