1123794 - Give access to engineers to staging/prod infernyx

Status: RESOLVED FIXED

(Content Services Graveyard :: Tiles: Ops, defect)

Description

[Olivier Yiptong [:oyiptong]]

Please provide access to emtwo and maksik to the infernyx machine.

emtwo (Marina Samuel)
maksik (Maxim Zhilyaev)

are engineers working on a new feature for tiles, Related Tiles.

There is an experimentation component to the project, so they need to access raw data to produce conclusions.

Comment 1

[maxim zhilyaev]

Few notes on data to be computed on infernyx machine and moved to redshift database.
There are 3 data sets needed to filter our statically insignificant site co-occurrences:

- aggregated count of occurrences for each site in new-tab views
- aggregated count of co-occurrences for site pairs that show up together in new-tab views
- aggregated count of history occupied tiles across new-tab views
 
These sets will be computed via map-reduce jobs running on infernyx.
The resulted datasets will be moved over to redshift data-warehouse for subsequent access.

The number crunching algorithm that filters out noisy co-occurrences is described here: https://docs.google.com/a/mozilla.com/document/d/1o5DB-OFABV0Ze9ye9ve3gyBs-VHIsaLZtFDG58MQoKg/edit#heading=h.xwjr9eu9xn5c

Comment 2

[Olivier Yiptong [:oyiptong]]

> why do they need real data to write jobs, can't we do sample data or generate data
The first task is collected from a sample: 50% of the beta population as part of a telemetry experiment. We will be running experiments periodically, potentially the same experiments or different ones.

> are these ad hoc jobs?
The first task is ad hoc, with the idea of finding what tasks we'll run systematically.
One aspect is to have  experimentation phase, then when we find something that seems promising, we bake it in the product.

> duration of access
We'd like to give indefinite access to these engineers. We are building product features. Marina and Max are now putting 100% of their efforts on the Tiles project, being on another project before then.
We'll have a bunch more system and feature development after this experiment data analysis subproject.
For instance, both will be working on the "Related Tiles" project, which is end-to-end.

Comment 3

[Daniel Thorn [:relud]]

this is ready, pending review, merge, and deployment of https://github.com/mozilla-services/puppet-config/pull/1043

Comment 4

[Daniel Thorn [:relud]]

This is complete