1506677 - Convert one day of Heka-protobuf main pings into newline-delimited json

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

We want 1 day of data from the telemetry pipeline copied into GCP for performance testing.

The basic procedure will be the following:

* Use the moztelemetry API to read an RDD[HekaMessage] and write out a RDD[JSON]
* Store data in temporary staging area
* Copy data over to GCS


The day will be a weekday to get a reasonable sized load. In addition, it may be useful to generate 90 days of nightly data in the same format for further testing.

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1506677 - main ping conversion (Scala)

This converts a day of data for 20181101. It takes 9 hours to process, resulting in a 1TiB dataset with 160 partitions. It is stored in the following s3 location:

> s3://net-mozaws-prod-us-west-2-pipeline-analysis/amiyaguchi/20181101-main-json-gzip

Comment 2

[Anthony Miyaguchi [:amiyaguchi]]

The data is now also available under the following bucket:

> gs://bug-1506674/amiyaguchi/20181101-main-json-gzip/

Comment 3

[Frank Bertsch [:frank]]

Comment on attachment 9024842 [details]
Bug 1506677 - main ping conversion (Scala)

LGTM