1298120 - Separate Aggregation functions into a utility library

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Ryan Harter [:harter]]

A strange Spark bug forced us to include some cross sectional specific functions and general statistical functions in the same package. These statistical functions are going to increase in size, so we should try to decouple these.

Comment 1

[Ryan Harter [:harter]]

I owe a write up on this error, but it turns out that this bug disappears when we include the SQLContext initialization in the main function. I took a crack at decoupling these packages today, but hit a new and exciting error. 

In short, we'll need to move generateCrossSectional into the CrossSectional class which will be easier once Bug 1298123 is closed up.

Longer explanation:
https://github.com/harterrt/telemetry-batch-view/commit/96267ec2c57bef5cf1f97d0bc0e95f53e3622479

Comment 2

[Ryan Harter [:harter]]

Promised write up:
http://blog.harterrt.com/strange-spark-error.html#strange-spark-error

Here's an example of what this pull request will look like:
https://github.com/harterrt/telemetry-batch-view/commit/fee2ad625e11011b1083f2c57216023ef8b5202d

Comment 3

[Ryan Harter [:harter]]

Attachment: https://github.com/mozilla/telemetry-batch-view/pull/109