1337044 - Add retention dataset to telemetry-batch-view

Status: RESOLVED WONTFIX

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P3)

Description

[Anthony Miyaguchi [:amiyaguchi]]

The churn/retention dataset currently exists at telemetry-parquet/churn/v2 and is generated by a jupyter notebook. [1] It has a few uses including work for search and stub attribution.

This job should be written in scala for consistency with the other broadly used datasets. There should be some suite of unit tests that validates expected behavior, and documentation that supports its design and use case.

Since this job is also being migrated, there should be some sanity check that the old and new versions are generating the same data. A jupyter notebook in bug 1329842  supports the transition in a rollup that uses python + redshift and scala + spark. A similar report for validating change can be made for retention.

[1] https://github.com/mozilla/mozilla-reports/blob/master/etl/churn.kp/orig_src/Churn.ipynb