1379806 - Evaluate macrobase-lib on `crash_rates` dataset

Status: RESOLVED WONTFIX

(Data Platform and Tools :: Monitoring & Alerting, enhancement, P3)

Description

[Anthony Miyaguchi [:amiyaguchi]]

`macrobase-lib` is the reference implementation of MacroBase, writen in java and published on maven. [1][2]

The `crash_rates` dataset contains a set of attribute and metric columns related to the measurement of crashes over Firefox telemetry data normalized by session length. It currently is obtained using a 1% sample over 2 weeks of `main_summary/v4` data.

Spark contains all the pre-processing tools necessary to extract and transform a dataset from our data lake, and then load a mysql store. MacroBase contains an ingestor for a single csv file, but could be modified to read from partitioned csv files or through a database via jdbc.[3] The results of batched and windowed detections and explainations can be computed and stored as a reference for spark-streaming computations. To measure performance, the hardware configuration and application execution timings should be collected. 

Results should be reproducible. Since MacroBase maximizes resource usage on a single machine, the entire dataflow can be captured in a container. It should be simple to rerun a modified pipeline (new attributes, metrics, time window, batched vs windowed) with credentials to access `s3://telemetry-parquet/main_summary/v4`. Java libraries are compatible with scala and can be written in a notebook format, publishable to RTMO.[4]


[1] https://github.com/stanford-futuredata/macrobase
[2] https://mvnrepository.com/artifact/edu.stanford.futuredata/macrobase-lib
[3] https://github.com/stanford-futuredata/macrobase/blob/master/contrib/src/main/java/macrobase/ingest/CustomJDBCIngester.java
[4] https://github.com/mozilla/python-zeppelin

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

Attachment: Bug 1379806 - Add instructions for running MacroBase GUI #2