1450079 - Add option to align number of input partitions to default parallelism

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

Currently the default behavior is to read partitions in block sizes of 256 MB. This does not work well when processing a small amount of data on a large number of workers. For example, processing a dataset that is under several GB will leave most of the available workers with no work.

To get around this, we should add an `--alignment` option to MainSummary, which will set the number of input partitions to a multiple of `sc.defaultParallelism`.

The performance of aligning the number of partitions to default parallelism was explored on Jan 16 in the `Message.toJValue` PR in moztelemetry.[1] Over the course of a single day with the same number of workers, the aligned jar took 3 hours as opposed to 3.4 hours. This change has a minimal gain for large datasets, but this is more efficient for smaller datasets. 

Using this option is strictly better the naive partitioning method if tuned correctly due to higher worker utilization.


[1] https://github.com/mozilla/moztelemetry/pull/15

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Correction: the default is 268 MB, which is 2^31 bits.

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/telemetry-batch-view/pull/406

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

This ended up being implemented as two options.

> --read-mode (fixed | aligned)
> --input-partition-multiplier <int>

`fixed` is the current behavior, reading in files until partitions are filled. `aligned` will use evenly-spaced partitions aligned to the default parallelism.