1321414 - Create v2 desktop churn/retention dataset in telemetry-parquet

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The churn notebook currently dumps gzipped data to s3 in two separate buckets [1][2]. In order to make the data eventually available to re:dash, it should be stored in the telemetry-parquet bucket as a parquet dataset.

There is already a churn dataset located in `telemetry-parquet/churn/v1`, corresponding to a view generated via telemetry-batch-view. These two datasets are not necessarily the same, but I don't see any harm in storing this as `telemetry-parquet/churn/v2`.


[1] s3://net-mozaws-prod-us-west-2-pipeline-analysis/mreid/churn/
[2] s3://net-mozaws-prod-metrics-data/telemetry-churn/

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Github pull request

The attached pull request will contain the relevant changes for this bug.

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Final pull request on mozilla-reports

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

The churn notebook now lives in `mozilla/mozilla-reports/etl/churn.kp`. It has been modified to read the current date from the environment, and will upload parquet to `s3://telemetry-parquet/churn/v2`.