1386569 - Write a one-off jupyter notebook to aggregate RAPPOR experiment data

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Alessio Placitelli [:Dexter]]

Once the RAPPOR SHIELD study is deployed (bug 1386566), we will need to write a one-off Jupyter notebook that reads the generated data and produces an aggregate view of it. At the very least, the notebook should:

- perform the aggregation;
- make the data available in a private S3 location;
- make the aggregated data easy to navigate and understand.

Comment 1

[Florian Hartmann]

We have developed two Jupyter notebooks: One for prototyping [1] and another one that is meant to be used in production [2].
The differences are as follows:

- The prototyping notebook contains all components that are potentially interesting for use with RAPPOR. This means it includes a lot of code that is optional. For some problems it also contains competing solutions
- The prototyping notebook works using the datasets generated by Google's repository [3, 4]
- Generally, the prototyping notebook contains a lot more explanations on how various things work
- The production notebook only contains the components that worked best for our use case. This makes it a lot smaller and also more straight-forward to use
- The prototyping notebook can optionally generate datasets using Spark. Alternatively, the user has to import their own dataset into the specified format

[1] https://github.com/florian/rappor-server/blob/master/RAPPOR-Prototyping.ipynb
[2] https://github.com/florian/rappor-server/blob/master/RAPPOR-Production.ipynb
[3] https://github.com/Alexrs95/rappor
[4] https://github.com/google/rappor

Comment 2

[Florian Hartmann]

I transferred the repository Alessio's GitHub account. It can now be found here: https://github.com/Dexterp37/rappor-server
We'll move the repository to the Mozilla organization later on.