1125451 - Add support for the new v4 ping data format in Spark

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P3)

Description

[Katie Parlante]

- rvitillo has been exploring spark (http://robertovitillo.com/tag/spark/); integrate this into the pipeline
- Implies a similar db-backed table of datastore filenames for filtering (don’t want to list S3 every time - too slow)

Comment 1

[Roberto Agostino Vitillo (:rvitillo)]

Katie, I am not listing the filenames manually, I am using mreid's db-backed service (http://ec2-54-203-209-235.us-west-2.compute.amazonaws.com:8080/files).

Comment 2

[Mark Reid [:mreid]]

(In reply to [e-mail] from comment #1)
> Katie, I am not listing the filenames manually, I am using mreid's db-backed
> service
> (http://ec2-54-203-209-235.us-west-2.compute.amazonaws.com:8080/files).

Right, this is a reminder that we should do the same for the new data on S3 too.

Comment 3

[Katie Parlante]

Bug triage comments: Looks like Pipeline team can't do this in Q1.

Comment 4

[Roberto Agostino Vitillo (:rvitillo)]

Mark, can you can provide me with a python module that given a file that uses the new data format returns a list of strings with the payloads? If so I should be able to integrate it easily with Spark.

Comment 5

[Mark Reid [:mreid]]

Here is some code that reads the new data format - it returns the actual Heka message objects. The JSON data is in a field called "Payload", though some of the other message fields are likely to be of interest as well.

There is probably no reason to include the "raw" or "header" pieces when integrating with Spark.

Comment 6

[Mark Reid [:mreid]]

https://github.com/mozilla/telemetry-server/blob/heka_compat/telemetry/util/hekastream.py

Comment 8

[Roberto Agostino Vitillo (:rvitillo)]

I don't see this as a duplicate: this bug is about adding support to read Heka objects in Spark while Bug 1152539 is about adding a client API to Spark.

Comment 9

[Mark Reid [:mreid]]

Ah, sorry. I thought both were about reading Heka data in Spark.

Comment 10

[Roberto Agostino Vitillo (:rvitillo)]

The Spark API supports now v4 submissions, see http://nbviewer.ipython.org/github/vitillo/emr-bootstrap-spark/blob/master/Telemetry%20Hello%20World%20%28v4%29.ipynb