1476067 - Set sane defaults for main_summary DAG when the deploy environment is dev

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

When adding a new dependency to the main_summary DAG, we are generally looking that the data dependencies are set up correctly and that the job will run without any serious errors.

The full main_summary DAG is run when debugging locally. Instead, a subset of the main pings should be selected, and the size of clusters adjusted for the smaller volume.

It might be useful to have a macro or function that decides on the size of the cluster for us, based on the linear scaling of performance vs cluster size. For example, if the job originally used 30 clusters and now only uses 1, downstream jobs could use the ceil(nodes/30). 

In practice, everything will probably run on a single machine when DEPLOY_ENVIRONMENT=dev.

Comment 1

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-airflow/pull/326

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

The PR describes the changes that were involved for setting sane defaults. One thing that I ended up doing in the Dockerfile that might relevant for reference is setting `EMR_INSTANCE_TYPE=c3.2xlarge`. 

It takes roughly 2 hours to run the entire pipeline. There is quite a bit of cluster startup overhead involved with this process.